Java Bien!

For two years I’ve been working at Algodeal. Two great years full of fun and challenges, going back to full-time developer after 4 years of technical consulting and agile coaching. Being the CTO at Algodeal, I thought that hiring a small team of talented people and being an equal part of this development team would be fun. It is! It worked for me at Adesoft. It worked again.

If I had to do a technical retrospective, here’s what I’d say:

There are great tools for java developers out there. Want to know my top 4?

Git: I’ve said it millions of times this year: you have to learn git. No excuse. Using a different VCS would really hurt now. Using git everyday for 2 years was also a good reason to make a lot of presentations to various companies, users groups and conferences. (Even more to come later the year) If you need to be any more convinced, go check out how git bisect could save your day.

Mockito: The other tool dear to my heart. Other mock framemorks made it clear that I should stay away from mocks and stubs, except for custom-made ones. Now, this all changed. I’m pretty sure that this kind of tool can make your tests quite redeable and super easy to refactor. A mindset change is all it takes. I got rid of most custom stubs, fakes and builders I used to write. Slides in french.

Google Collections, aka Google Guava: Ever wanted to write pseudo functional code in Java? This is easy with Functions and Predicates and a little bit of glue in between:

List<String> months = Arrays.asList(
    "January", "February", "March", "April", "May", "June", "July", "August",
    "September", "October", "November", "December");

String juneToJuly = with(months).only(startsWith("J")).exclude("January")
    .to(UPPERCASE).join(" to ");

assertThat(juneToJuly).isEqualTo("JUNE to JULY");

Nice isn’t it? Guava is full of other surprises, Suppliers, Futures, Preconditions, to make code faster and more reliable.

Infinitest: At first, there was JUnitMax, written by Kent Beck himself. An eclipse plugin that would run the full test suite for every change you make in the code base. It really changed the way I TDD. I loved this tool, I talked about it to everybody, until Kent Beck decided to discontinue the tool (guess what, one year after, he changed his mind). Enters Infinitest: better, faster, stronger. If you do TDD and don’t use a continous testing tool, you really miss something fundamental. Tools are mature enough for you to get onboard.

Team of 5 or 6 is perfect size:

It’s incredible how fast a small team can go. I’ve been in bigger teams and it’s so hard to keep everybody informed, and up to date. It’s so hard that you don’t do it. A team of 5 is made of enough opposite points of view and yet is quite easy to focus as a one. Sure a few expertise remain not shared by everyone, but with a team this size, everybody knows what he doesn’t know. And when it becomes depressing, he learns. In a large team, you don’t even know what you don’t know.

Build and test the whole thing fast. I mean fast:

This one I’m very proud of. From the beginning, we took good care of our test suite and tried to keep it as fast as possible. By fast, I mean under 4 minutes for the full build, unit tests, integration tests and functional tests. Guess what is the biggest benefit you get for free, trying to keep a quick build? You gain to challenge any business and technical decision: Will it be easy and quick to test? Will the tests be stable (anybody testing ajax)? If the answer is NO, then is the feature really useful for the users? Can it be done a simpler way. Turns out that if it can be easily/quickly tested, it can be easily/quickly developped, easily/quickly deployed and easily/quickly used.
Not always true of course, but often enough for this to become one of our hard rules.

It even lets us do unbreakable serverless continous integration. Wanna try?

Challenging the architecture often is a good habit:

It forces you to choose an architecture easy to deploy and easy to get rid of. Eg. We got rid of Voldemort in one day. That was super easy since it’s API is made of thow methods get(key), put(key, value). Compare that to JDBC API. If you want the full story, the data that we now store on HDFS, was first stored on a local filesystem, then on a remote filesystem, then on Voldemort.
An architecture that you cannot refactor or get rid of easily might not be worth setting up in the first place.

Hope this retrospective will help you and your team. Any feedback is welcomed.

Hier soir, Octo recevait presque tous les speakers qui animeront une présentation lors de la conference USI 2010 les 1 et 2 juillet prochains. Tout d’abord, merci à Octo pour cette soirée très sympathique et plutôt drôle.

Université du SI

L’occasion était donnée a chaque speaker de présenter son sujet en une minute. Toutes ces minutes, mises bout à bout, feront un petit film de présentation des sujets en une heure. Ce film vous permettra de composer votre programme autrement qu’en lisant une page web ou un programme papier. Bonne idée non ?

Voici mes coups de coeur :

• Keynote d’ouverture de Chris Anderson, éditeur en chef du magazine Wired. Son point de vue sur les évolutions de l’économie numérique en général et de la presse en particulier sera certainement enrichissant.
• Ma présentation GIT une heure plus tard. Celle là, vous n’avez pas le droit de la manquer, je le prendrais personnellement…
• L’inratable “Why, not how” par Neal Ford et Martin Fowler. Superbes speakers, sujet captivant.
• “The future of money” par Renier Lemmens, PDG de Paypal. Il y a là une réelle transformation déjà amorcée, il sera intéressant d’avoir le point de vue d’un acteur de poids dand ce domaine.
• “Avaler la pilule rouge, recracher la pilule bleue” par Guillaume Duquesnay, coach Octo.
• Puis “L’histoire de l’informatique” par Philippe Nieuwbourg, Directeur du musée de l’informatique.
• Et pour finir, une belle keynote de clôture par Juan Enriquez : l’Homme va-t’il prendre le contrôle de son futur ?

Enfin, ne soyons pas langue de bois (pas mon genre), la keynote de Bernard Stiegler du mardi matin devra être bien meilleure que sa présentation en 1 minute, soporifique à souhait.

Une bien belle conférence pour geeks et boss se prépare. J’en salive d’avance.

If you are a little involved into writing web applications, you have to know Yahoo!’s YSlow and Google’s PageSpeed. These are two Firefox plugins to help you accelerate a web site.

One thing they will help you do is to reduce the size of all those png images to decrease bandwidth. Sure png images are lighter than gifs but still you can remove some fat out of them keeping quality constant. Take a look at Smush.it if you want to learn more.

There are tools that can do similar things on the desktop. Pngcrush is one of them. This is as simple as this:

pngcrush imageIn.png imageOut.png

or even better:

pngcrush -rem allb -brute -reduce imageIn.png imageOut.png

Now how can we automate that? If pngcrush was java-based, you could do it dynamically with a servlet filter or at build time using ant/maven. A more effective way to do it, is at commit time using git.

The idea is to crush each png image added to the project the first time it is committed or each time it is updated. Git can do that with a simple hook script that will be executed pre-commit.

#!/bin/sh
if ! which -s pngcrush
then
  echo "Please install pngcrush to reduce png images size before commit"
  exit 1;
fi

for file in `git diff --cached --name-only | grep ".png\$"`
do
  echo "Crushing $file"
  pngcrush -rem allb -brute -reduce $file ${file%.png}.new | grep "filesize"
  mv -f ${file%.png}.new $file
done

Get this code on Gist.

Have fun!

The day maven ruined my day (again…)

Yesterday, I lost 2 hours because our Maven project at Algodeal wouldn’t build anymore. I couldn’t execute:

mvn eclipse:eclipse

nor run the full build.

With the help of a colleague, we found out that only a two steps build would do the trick:

mvn clean install -DskipTests;mvn eclipse:eclipse

Obviously maven is already not my best friend but yesterday I was just fed up.

How git is called to the rescue
Read more…

“Why use a continuous integration server?”

That’s the question we ask at Algodeal. Having spent years preaching for each team to use a CI server, we installed and used Hudson since the very beginning of the project. However, it’s been months since anybody looked at the Hudson dashboard. Every commit breaks the build. And you now what? Nobody cares

How did it happen?

The key problem is the time wasted maintaining and Hudson server. Every brick mandatory to the build needs to be installed on the server. Our product relies on (unfortunately) latex, mono and a few R packages. The CI server needs to be up to date always, rebooted sometimes, cleaned more often than never.

Why spend time maintaining a CI server when we could make sure the tests pass locally before pushing to the shared repository?

Decision made: Small aware team + quick build = Bye Hudson! And don’t even take a single second to unplug it.

We’ve got a situation

As time goes by, as tests are written, a build gets slower and slower. One minute, then two, then five. Running all the tests locally becomes a pain. It’s tempting to push to a CI server without running the tests before and wait for the results drinking a coffe. Here’s the choice: either stare at maven for 5 minutes or push broken code.

Arms race

We don’t want to push broken code. We don’t want to waste time staring a the tests. So should we setup a two phases commit with two CI servers? For example TeamCity proposes a remote private build feature which guarantees an unbreakable build timeline. This looks like an arms race lost in advance. Soon we would have a build farm to maintain. Scary!

Reduce the build duration

Let’s do a root cause analysis instead of putting money and time on a build farm. Let’s quicken the build. After a few days, some black magic (a future post), our build was reduced from more than 5 minutes to less than 2 minutes. We can breath again. The main question however still remains. How not to use a CI server in the long term and not suffer the increasing time lost staring at the builds locally?

Here comes Git

At Algodeal, we choose git a year ago. Its stability and ease of maintenance are strong advocates. Even if daily usage is a bit odd given the bad front-ends available.

Reading “Git Magic“, I realized it would be so simple to setup a private local build similar to TeamCity’s. Without any server! We’d clone our working directory, run the build from the clone and commit only in case of success.

With a different version management system, we’d face two problems: How long to duplicate the whole project if it becomes large? Once the changes are pushed to the repository, how does the working directory “knows” it’s been commited?

With git, it’s a piece of cake. First, we ‘git clone’ the working directory to another folder. Git does the copy *very* quickly. Next times, we don’t need to clone. Just tell git get the deltas. Net result: instant cloning. Impressive.

What about the consistency? Doing a simple ‘git pull’ from the working directory will realize, using delta’s digests, that the changes where already pushed on the shared repository. Nothing to do. Impressive again.

Of course, while the build is running in the second directory, we can keep on working on the code. No need to wait.

The ultimate private build?

We now have a private build with no maintenance, no additional installation, not dependant on the IDE, ran with a single command line. No more broken build in the shared repository. We can recycle our CI server.

Yes. You’ve heard well. We’ve just built a serverless CI. Every additional feature of a real CI server is noise to me.

The Recipe

For the more curious of you, here is the script:

#!/bin/bash
if [ 0 -eq `git remote -v | grep -c push` ]; then
  REMOTE_REPO=`git remote -v | sed 's/origin//'`
else
  REMOTE_REPO=`git remote -v | grep "(push)" | sed 's/origin//' | sed 's/(push)//'`
fi

if [ ! -z "$1" ]; then
  git add .
  git commit -a -m "$1"
fi

git pull

if [ ! -d ".privatebuild" ]; then
  git clone . .privatebuild
fi

cd .privatebuild
git clean -df
git pull

if [ -e "pom.xml" ]; then
  mvn clean install

  if [ $? -eq 0 ]; then
    echo "Publishing to: $REMOTE_REPO"
    git push $REMOTE_REPO master
  else
    echo "Unable to build"
    exit $?
  fi
fi