Increment on the long term, iterate day by day

this is the transcription of the 10-minutes lighting talk i had last weekend during our company meeting in Amsterdam, which is based on the great article by Alistair Cockburn “Incremental means adding, iterative means reworking”. so, here it comes..

Incremental vs. iterative development

every project is big enough not to be completed just in one shot. so, we focus on delivering subsystems on a regular basis, e.g. one or two weeks. we face complexity with simplicity and feedback.

first strategy we can adopt is delivering complete subsystems, every week, splitting projects into small chunk of complete functionality, also known as user stories. we add features every other week: this is incremental development, the heart of every agile process.

in other words, we try to get it right in the first place, targeting quality for a final product.

another strategy is reaching the final product by revisions. we start from a rough prototype, and improve it by later adding details, till the final result. we do reworking: this is iterative development.

a great essay on the two strategies is Jeff Patton’s Monalisa example, as reported by Gojko Adzic on the waterfall trap.

back to the initial motto, here is my suggestion: even if you’re already doing incremental development, don’t forget to iterate. (it’s a shame the term “iterations” does not relate to iterative development, but to incremental development!)

why? because customer, not us, choose when to stop incrementing (adding) and iterating (reworking): you’ll be surprised by what a customer can consider good enough.

Time for an example

my team is currently working on an e-commerce website, and the customer asked for a complex checkout procedure: adding items to the shopping cart, mandatory login, shipping and billing addresses, payment details (e.g. credit card info), and finally a checkout with notification via email and to the reporting system.

during release and iteration planning, we first asked the customer to split the procedure into smaller user stories, so that we could understand and estimate with confidence. then, we started developing the most important ones; for example, no billing address nor reporting system notification were chosen.

this is incremental development.

one day was the turn of the shipping address user story: calculate shipping costs for each country (Italy has a fixed cost, other countries costs are taken from the database of our customer’s shipping company), displaying italian provinces when shipping to Italy, validating zip codes when in Italy (CAP). everything in a dynamic fashion.

having a lot of work to do, we decided to start with a minimal layout: a combo box for countries and a text displaying shipping costs for current country. it was far away from the final product, but gave us feedback that everything was working, from UI to database import script.

later, we decided to put two forms, for Italy and other countries, into the same page. that was an easy way for having a different layout and data for Italy. again, it would have probably not been the final product, but helped us focusing on designing UI and setting up data.

we finally moved on trying to understand how to dinamically change layout, based on current country. first idea was to use Javascript code client-side: something like having two hidden forms from which setting a third form’s innerHtml, and the like. we did a spike, but our result was not satisfying.

then, having our iteration going to end soon, we decided to propose to the customer a simpler way: an onchange event on the country combo box! dynamic, but server-side. it took one pomodoro for spiking a solution. then we just asked the customer, and after other two pomodoro of layout restructuring, it was done.

that was iterative development.

hope it does make sense to you: increment on the long term, iterate day by day!

ATFQ

after almost a full day of nervous coding and testing..
trying to simplify a complex scenario..
before making it more complex with new requirements..
i finally found out it was easier than i thought..
simply asking the customer what he wanted.

so, please, don’t forget to ask the f**ing question!

Say what you want

did you ever want to be notified about latest build status? think about having a script that monitors your build server via RSS feed, and notifies success or failures saying something like “build broken for project XYZ”. add a rubysh taste, and…

i’m happy to announce cruise-monitor has just been published! as README says:

“Cruise-monitor is, well, a monitor to CruiseControl build status, via RSS feed. It uses MacOS ’say’ command for notifications. So far, only CriseControl.rb is supported, but plans are to support CC and CC.NET as well”.

this is the first open-source project hosted on our company public servers that i’m involved in. it basically was born after a few broken builds for the project my team is currently working on: a Rails application built on a CruiseControl.rb continous integration server.

it happened a few times no one noticed a failing test or a missed svn add. we started using a feed reader playing sounds on each new build, but still it was not able to distinguish failure from success. that’s when we thought about using great MacOs say command.

it started as a spike on ruby, RSS and system exec, but it soon turned into a real project: a Rake build file; unit, integration and acceptance tests; README, LICENSE and TODO files.

so far i’m the only committer! i hope the community around will grow a little bit. i’m going to add details on our public confluence in the following days.. so, stay tuned!

There’s more than one way to skin a cat

iterative and incremental development means delivering functionalities one at a time driven by some priority critera (e.g. business value). similar functionalities happen to be developed not all at the same time. so, design focus on supporting all the functionalities planned for a given iteration, and nothing more. key to succeed are simplicity, flexibility and a wide automatic test suite.

imagine a financial project where customer needs to process via nightly batch all monetary transactions received daily. he ask for supporting critical calculations first, and set at lower priority to automatic double check (it can be done manually in the meantime). in a plan of a few months, 3 or 4 different calculations are going to be developed.

but we start from the first one, and let the code (aka design) support the others later.

just after the first user story get signed, design consists of three main roles:

• a BatchCommand that collects items and process calculations, and then updates state on items (e.g. date processed)
• a Gateway to the database that knows criteria to filter items and knows how to select and save items from/to the database
• a Connector in charge of handling connection to the database

here’s a brief recap using CRC cards:

next iteration starts, and customer select another calculation to be developed, which just differs in selection criteria. then, the team has to find out the simplest way to support this scenario. well, a few exists, of course. in this post, i’m focusing on some design strategy we can use, comparing them.

we need to change Gateway selection criteria. someone says go for inheritance, and override criteria in a subclass. someone else says use composition, and move responsibility to change into another object. you choose, me too: composition!

so, we change Gateway to delegate a collaborator to gather selection criteria: let’s say, a CriteriaBuilder, even better as an interface. with our favourite IDE we perform an “extract method” refactoring followed by “move method”, moving selection criteria to a criteria builder. then, we add another builder for current user story. finally, it’s just a matter of wiring two Gateways with two CriteriaBuilders. design ends up being like this:

what’s wrong with this? well, nothing i would say, dependencies are well handled: low-level details (Gateway) depend on abstraction (CriteriaBuilder interface) and high-level policies (concrete CriteriaBuilders) don’t depend on anything, they provide that abstraction. in other words, we succesfully applied Dependency-Inversion Principle. so, again, what’s wrong?

we created two different gateways, that’s the matter, in my view. we stated Gateway is an infrastructure detail, it “knows how to select and save items from/to the database”. so, why do we need two different ones? has infrastructure changed? not at all.

in OO an object has an “outside” and an “inside”. GoF, in the Decorator pattern, to explain this simple rule used the metaphor of an object skin and guts:

“Changing the skin of an object versus changing its guts. We can think of a decorator as a skin over an object that changes its behavior. An alternative is to change the object’s guts. The Strategy pattern is a good example of a pattern for changing the guts.”

now, imagine we move selection criteria “outside” from Gateway, into another object which collaborates with the Gateway, but doesn’t hold a reference to it: it wll be sent as a parameter of the message “please, apply selection criteria on this gateway”. let’s say we name this new role a Selector. we end up having one Gateway and two Selectors, this way:

that’s it.

to recap, my suggestion to you while doing design: whenever you find an opportunity to simplify object responsibilities, moving some of them away, ask yourself “is this a domain or an infrastructure issue?”. if possible, avoid mixing the two: leave objects cohesive, and apply DIP. then, all other things being equal, avoid having more instances of a given infrastructue class, unless you really need to set different infrastructures (in this example, a gateway to filesystem or an in-memory one).

as Beck says, “Design is beneficially relating elements”.

It seams open

i can clearly remember when i first discussed DIP and OCP with others: it was two years ago, during my apprenticeship as an XPer. to me, it was nothing new, i already had studied all the principles that now come under the SOLID ancronym. but, probably, i hadn’t digested them enough: something that just came later with experience.

Dependency Inversion was for sure my favourite since then, for its multiple implications depending on what “high level” and “low level” mean; in my view, there are at least two meanings: abstraction (high level policies vs. low level details) and layering (close-to-user layer such as GUI vs. infrastructure layer). much more, i loved its love-hate with Dependency Injection (maybe more on this in a separate post).

Open-Closed is harder to understand, at first. how could you “change without modifying”? abstraction is the key! let modules depend on abstraction, then provide new implementations when behaviour has to change, without the need to modify existing code. in other words, always depend on abstraction: which, in the end, is DIP itself. OCP and DIP are such “yin and yang” in software design: they help achieve each other.

then, when i first discussed OCP with team-mates, i pushed for an analogy with Feather’s Seam Model. it’s discussed in “Working Effectively with Legacy Code” book: use seams to let legacy (which means untested) code be tested. to be fair, my analogy was not welcomed too much! i had to force my thesis a bit, and in the end not everybody was convinced.

two years later, it happened again! indeed, a few months ago we had a study group on OCP. i was in charge of preparing material to study on, and i chose a few corollary articles: first chapter from GoF Design Patterns book, which focuses on “design targeting interfaces”, and WELC chapter 4 “the Seam Model”. this time i was more convincing, and the analogy between OCP and the Seam Model became clear during our study session! and now, i want to tell you too.

after reading the bunch of articles i prepared, i asked a colleague to state in a few words what OCP was about. he said “change behaviour without modifying code”. great! then, i asked him again to state what the Seam Model was about, and he said “let code behave in a different way without modifying it”. well.. nothing left to say!

abstraction is the key, that’s true. but what about code which doesn’t follow OCP/DIP? it doesn’t depend on abstraction. we can modify it, refactoring, but we need an automatic test suite, in order to guarantee no behavioural change. and that’s exactly what Feather’s model is about: change code a little bit putting seams, to test it in isolation.

on the other side: what seams can you use to test OCP-compliant code? of course, already existing abstractions: you just have to change enabling points (in a test, usually in fixture setup).

in the report for our study session, on our internal wiki, we wrote:

We then discussed what OCP and Feather’s Seam Model have in common:

• they seem the same idea, applied to reach different goals
  • OCP: put abstractions to isolate from future source code changes
  • Seam: put abstractions to test applications without changing its source code
• to recap
  • closure/abstraction = seam
  • “main() routine”/factories = enabling point

to be precise, Feather’s model is about three different techniques, useful for testing legacy code written in any language, not just object-oriented ones. he talks about preprocessing seam, linking seam and object seam. so, the analogy with OCP is just between abstractions and object seams, even if sometime linking techniques are also used to achieve abstraction (such as reflection or some configuration-based IoC tool).

so, when i watched Misko’s clean-code-talks videos, i was surprised to hear him use the term “seam” while talking about DI and SOLID principles: he confirmed my analogy have sense!

Retrospective hints

i was thiking of posting some thoughts about last retrospective my team just had. funny enough, today i opened my feed reader and found this post by Jay Field on exactly the same topic: let the retrospective be driven by an external facilitator, and do a safety check before starting.

last friday we had a team retrospective. doing project ones would now be impossible, while we’re working on 3/4 projects and rotating pairs each day (more on this later). moreover, this time frame (last few weeks) has been very hard for us all; you know, the “rise and fall”: hard-to-folllow customers, projects coming and leaving, zero-velocity iterations, a few bugs.

yeah, but we also had a great surprise: we now have a new UK managing director! having Marco in Milan last week, we proposed him to drive our retrospective.

he chose to start the meeting with a “ballot”: write down on a note how likely you are talking to others, on a scale from 5 to 1. i never tried this strategy, even if Piero and me talked about during our session at Italian Agile Day. it was really interesting, numbers weren’t fully as expected. good: feedback gathered.

we used the starfish form, also known as “start-doing, stop-doing, keep-doing, more-of, less-of”. this time, after a rough clustering, we focused on “good and bad” for each cluster, and then proposed actions to support the good or to avoid the bad. then, we performed a quick vote and chose rule champions. (i’m the champion for “shared pomodoro”… let’s see…)

to recap, what went well:

• we asked an external person to drive our retrospective; of course, he knew what a retrospective is and how to drive it. but he really didn’t know what our issues were. nevertheless, he was able to understand when a bit more discussion was good and when to stop it
• everyboby knew what the feeling and trust were; so, core issues have been addressed, being both important to everyone and source of frustation for a few

now the hard part: on monday we’ll start adopting some very strict rule, everyone voted! as usual, we’ll try for one week at least, then we’ll reflect on data and feelings. let’s adapt!

Show all Java imports given a root package

here’s a simple shell script to collect all imports in Java source code files, given a root package.

find src -iname *.java \
| xargs grep -i "import another.project" \
| cut -d " " -f 2 | cut -d "." -f 1-3 \
| sort | uniq

results look like this:

another.project.domain
another.project.presentation
another.project.util

change “1-3″ to reflect how deep the filter should be.

thanx to Andrea for showing me cut command. i would have used my belowed write-only awk!

Chronicle of a (damn stupid!) bug

mom said “test everything that could possibly break”. now, how to define “everything”? well, the story goes like this…

last week, me and Antonio were pairing trying to find out what was causing a very strange behaviour in our codebase. a batch process, loading data from a file and populating an “ATM transactions” table, should then put duplicate records on a “duplicate ATM transactions” table. so far, so good, we worked on stories like this zillion times.

when our customer’s proxy finally run the batch on a PREPRODUCTION environment, something went wrong. she then collected a log file, from wich we started investingating where the problem could be. what a big surprise reading “unique constraint XYZ violated” for “duplicates” table! could be? we put no contraints at all on that table! even more, XYZ contrainst was declared on “transactions” table, not duplicates!

at first sight, there was nothing wrong in the codebase; we then tried to reproduce the bug on our DEVELOPMENT environment too, with no luck. so, we decided to log more debug infos, such as “executing INSERT INTO transactions” and “executing INSERT INTO duplicates”. test all, commit, deploy and go!

after another run, this time on TEST environment, bug was there again, but at least we got more log to read. can you see? yeah, debug info was “executing INSERT INTO duplicates”, followed by the silly “contraint violated” exception! gosh! how could that be?

support from customer’s internal technical staff didn’t help much, we just find out that there were “something” different from DEVELOPMENT and TEST/PRE environments. very strange. anyway, we compared genearted DDL for tables on both environments, but, again, nothing helpful.

then light came, thanks to our new team-mate Roberto, our little Oracle guru!

on our daily standup, we let the team know we were with no ideas, so Roberto proposed himself for a pairing session with me (the team is working on few projects, on which we turn pairs on a weekly basis). after reviewing together (useless) DDL scripts, he then started showing me some magic with Oracle “reflection” stuff: queries on metadata, such as tables, contraints, and so on.

i was talking with a team mate, who just asked me something, when i turned back on the desk, and found Roberto with a smily face saying “Jacopo, i know what’s wrong, but you’ve got to wait for our 5 minutes break!”. arghh! couldn’t really wait!

you know, the problem was in the only part we didn’t test: customer’s internal stored procedure invocations, for creating database object alias!

we use incremental SQL scripts to recreate from scratch the database structure: we just skip stored procedures for grants and alias, because they cannot be run in DEVELOPMENT environment. we delay that feedback to a manual run (a.k.a. a demo!) in TEST and PREPRODUCTION environments. and that’s exaclty what we had: feedback on our broken SQL scripts. one of the alias was wrong, duplicates table was pointing to transactions table! it was probably caused by a copy and paste from another script (and that’s the saddest part).

so, how to define “everything”? simple, everything!

My results

i’m 48 hours late. friday was the “results day”, as planned during the last Italian Agile Day. i’m one of the lucky ones, working in an agile team, both as a developer and a mentor too. but please, don’t think there’s no room for improving, nor for making mistakes: you’re wrong.

so, the bad things first.

i’m still too difficult to pair program with. i grab the keyboard from the driver’s hands not jut everytime my head has a new idea, but also when i see he/she’s doing a mistake. sometimes i “just” hit ESC from the navigator side whenever something wrong is coming. that’s truly, deeply, honestly wrong. i learned a lot from my mistakes, i should give a change to try (and fail too) to my team mate, in a timed-box fashion. even worse, my peer could have a simpler idea than mine, so why stop it? lesson learned: while acting as the navigator, tell your ideas first, and eventually grab the keyboard.

fine, now the good part!

one of the biggest risk on the project for our last customer was third-party services: API not yet ready, even functionalities unclear. for the mailing service, we were given an early WSDL to play with. it took a few days to develope a facade and an adapter to the system, in order to isolate our codebase from web services. even if it was incomplete (no way to send individual messages, just collective newsletters), we found out a simple way to send individual email: programmatically create a newsletter with a single recipient. managers told the team they asked for a complete API, “soon”. i think you can see… it never came, budget was over, time rushed. so, our simple but non optimized solution is now going to go in production, and it works. lesson learned: don’t anticipate change, it could take different paths than you thought, or it could never happen, at all.

then, i recently re-joined a previous project, after almost 5 months. so, my “fresh mind” helped the team spotting out some design smells. i’ve done a few code reviews and design spikes, checking out the codebase from scratch, and tried to let a “fatter” domain emerge, grabbing code from infrascture-poisoned areas. a strong re-packaging helped a lot, too. as a result, i put some note on paper, and took a nice screenshot of the packages structure. what followed were discussions and collective study sessions, during which we incrementally applied some of the hints, but better hints came out too, thanks to the whole team! design is now clearer and communicates better (even if we all know there’s a lot more to do). all the team members, also the newest ones, can now work on new user stories in a more confident way. lesson learned: focus on design.

and now, what’s next?

something i really missed during the last year is collective study sessions. i read and code a lot studying in my spare time, but i still miss gathering feedback from others. even if during a retrospetive the team decided to improve study sessions, i’m going to focus on timed-box collective brainstorming. in my view, design and refactoring are good candidates. that’s why i’ll push for an internal “Design Pattern Study Group”, or doing exercises from Wake’s “Refactoring Workbook” book.

being a team is tremendously different from being “a group of co-workers”. shared values, shared goals, respect, “we’re all in this together” approach. my suggestion to you, start thinking: am i working in a team or in work-group? feed a team culture, starting from values. i think results will come, soon: individuals and interactions over proesses and tools!

One year of presentations

thinking back to what 2008 brought, one of the biggest news is me doing presentations, both at public agile events and at company seminars.

indeed, this summer me and Tommaso were at ESSAP doing a session on “Customer TDD and FitNesse”. then, in november Pietro and I were at the Italian Agile Day, presenting “Retrospective, Stand-up Meeting and Daily Journal”. then, just before xMas we had a company-wide webinar with me presenting “Testing in isolation pt.1, dependencies faking”.

doing mentoring also means having workshops and presentantions with other teams, so this year i also had the chance of presenting TDD, OO principles and mocking tools.

and, last but not least, i’ll be presenting in a few weeks at next italian Alt.NET conference (hosted by Sourcesense – more on this soon) talking, again, about Customer TDD and FitNesse, this time providing examples in .NET (even if the main theme remains the same).

being a developer does not help in any way presenting technical stuff in public. of course, a solid unterstanding ot the topics is mandatory, but knowing and telling are two completely different things. that’s why, doing and doing again, i finally hope to learn presenting in a clear and effective way. and improve my english too.

audience’s feedback is tremendously important!