Java Bien!

Next version of Mockito, my favorite mocking/stubbing framework, will provide deep stubbing. This kind of test code will become possible:

@Test
public void canStubOneLevelDeep() {
    OutputStream out = new ByteArrayOutputStream();

    SocketFactory socketFactory = mock(SocketFactory.class, RETURNS_DEEP_STUBS);
    when(socketFactory.createSocket().getOutputStream()).thenReturn(out);

    assertThat(socketFactory.createSocket().getOutputStream()).isSameAs(out);
}

Notice the two chained calls in a when clause. Isn’t it cool? I think that’s cool.

I can make tests clearer in case of poorly designed production code. And who doesn’t write poorly designed code sometimes? I do more often than not. So why not have at least clearer tests?

It would be even cooler if we could get rid of the ulgy (very ugly?) RETURNS_DEEP_STUBS artefact and let the when clause sort things out on its own.

Hey, have you noticed I use Fest Assert, the best assertion library?

“Why use a continuous integration server?”

That’s the question we ask at Algodeal. Having spent years preaching for each team to use a CI server, we installed and used Hudson since the very beginning of the project. However, it’s been months since anybody looked at the Hudson dashboard. Every commit breaks the build. And you now what? Nobody cares

How did it happen?

The key problem is the time wasted maintaining and Hudson server. Every brick mandatory to the build needs to be installed on the server. Our product relies on (unfortunately) latex, mono and a few R packages. The CI server needs to be up to date always, rebooted sometimes, cleaned more often than never.

Why spend time maintaining a CI server when we could make sure the tests pass locally before pushing to the shared repository?

Decision made: Small aware team + quick build = Bye Hudson! And don’t even take a single second to unplug it.

We’ve got a situation

As time goes by, as tests are written, a build gets slower and slower. One minute, then two, then five. Running all the tests locally becomes a pain. It’s tempting to push to a CI server without running the tests before and wait for the results drinking a coffe. Here’s the choice: either stare at maven for 5 minutes or push broken code.

Arms race

We don’t want to push broken code. We don’t want to waste time staring a the tests. So should we setup a two phases commit with two CI servers? For example TeamCity proposes a remote private build feature which guarantees an unbreakable build timeline. This looks like an arms race lost in advance. Soon we would have a build farm to maintain. Scary!

Reduce the build duration

Let’s do a root cause analysis instead of putting money and time on a build farm. Let’s quicken the build. After a few days, some black magic (a future post), our build was reduced from more than 5 minutes to less than 2 minutes. We can breath again. The main question however still remains. How not to use a CI server in the long term and not suffer the increasing time lost staring at the builds locally?

Here comes Git

At Algodeal, we choose git a year ago. Its stability and ease of maintenance are strong advocates. Even if daily usage is a bit odd given the bad front-ends available.

Reading “Git Magic“, I realized it would be so simple to setup a private local build similar to TeamCity’s. Without any server! We’d clone our working directory, run the build from the clone and commit only in case of success.

With a different version management system, we’d face two problems: How long to duplicate the whole project if it becomes large? Once the changes are pushed to the repository, how does the working directory “knows” it’s been commited?

With git, it’s a piece of cake. First, we ‘git clone’ the working directory to another folder. Git does the copy *very* quickly. Next times, we don’t need to clone. Just tell git get the deltas. Net result: instant cloning. Impressive.

What about the consistency? Doing a simple ‘git pull’ from the working directory will realize, using delta’s digests, that the changes where already pushed on the shared repository. Nothing to do. Impressive again.

Of course, while the build is running in the second directory, we can keep on working on the code. No need to wait.

The ultimate private build?

We now have a private build with no maintenance, no additional installation, not dependant on the IDE, ran with a single command line. No more broken build in the shared repository. We can recycle our CI server.

Yes. You’ve heard well. We’ve just built a serverless CI. Every additional feature of a real CI server is noise to me.

The Recipe

For the more curious of you, here is the script:

#!/bin/bash
if [ 0 -eq `git remote -v | grep -c push` ]; then
  REMOTE_REPO=`git remote -v | sed 's/origin//'`
else
  REMOTE_REPO=`git remote -v | grep "(push)" | sed 's/origin//' | sed 's/(push)//'`
fi

if [ ! -z "$1" ]; then
  git add .
  git commit -a -m "$1"
fi

git pull

if [ ! -d ".privatebuild" ]; then
  git clone . .privatebuild
fi

cd .privatebuild
git clean -df
git pull

if [ -e "pom.xml" ]; then
  mvn clean install

  if [ $? -eq 0 ]; then
    echo "Publishing to: $REMOTE_REPO"
    git push $REMOTE_REPO master
  else
    echo "Unable to build"
    exit $?
  fi
fi

I’m a big fan of Google Collections. Functions and Predicates became my best friends to make Java’s syntax a bit functional-like. I like writing things like:

List<Contact> contacts = ...
List<String> toStrings = Lists.transform(contacts, Functions.toStringFunction());

and with the help of static imports:

List<String> toStrings = transform(contacts, toStringFunction());

If you use out-of-the-box Functions and Predicates, it’s a lot of fun. Now, let’s say, you want something more useful like:

List<String> names = transform(contacts, Contact.T0_NAME);

you need to accommodate with Java’s way of writing functions. That is write an inner class:

public class Contact {
...
public static Function<Contact, String> TO_NAME = new Function<Contact, String> {
    public String apply(Contact contact) {
        return contact.getName();
    }
}

It’s ok for one bean property, but writing functions for all of them is a lot of work.

What if we could write a JavaBean that would generate all these functions for us. Something like that:

public class Contact extends AbstractBean<Contact> {
public static final StringProperty<Contact> NAME = stringValue();
public static final IntegerProperty<Contact> AGE = integerValue();
public static final StringProperty<Contact> EMAIL = stringValue();

public Contact(String name, int age, String email) {
    super(NAME, name, AGE, age, EMAIL, email);
}

public String getName() {
    return get(NAME);
}

public Integer getAge() {
    return get(AGE);
}

public String getEmail() {
    return get(EMAIL);
}
}

Now, every property of my bean is also a Function converting a bean instance to it’s property value. To be clear:

Contact.NAME is a Function<Contact, String>
Contact.AGE is a Function<Contact, Integer>
Contact.EMAIL is a Function<Contact, String>

so that I can write:

List<Contact> contacts = ...
List<String> names = transform(contacts, Contact.NAME);
List<Integer> ages = transform(contacts, Contact.AGE);

Nice isn’t it?

Now, it can be even cooler. I can write something like that:

Iterable<Contact> david = filter(contacts, NAME.isEqualTo("david"));
... adults = filter(contacts, AGE.isGreaterThan(18));
... babies = filter(contacts, NAME.isEqualTo("baby").and(AGE.isLessThan(1)));

My bean properties are also Predicate factories!

With a custom utility class, I can even use a clearer syntax that Google Collection’s:

... ages = with(contacts).keep(NAME.isEqualTo("david")).to(AGE).list();

There are even more features. All AbstractBeans come with free equals(), hashCode() and compare().

The last neat feature is the ability to write a builder for any bean easily, with only one additional method:

public class Contact extends AbstractBean<Contact> {
...
static <V> BeanBuilder<Contact> with(BeanProperty<Contact, V> property, V value) {
    return new BeanBuilder<Contact>() {
        public Contact createBean(PropertyValues<Contact> values) {
            return new Contact(values.get(Contact.NAME), values.get(Contact.AGE));
        }
    }.with(property, value);
}

Now, I can write something like this:

Contact contact = with(NAME, "david").with(AGE, 34).with(EMAIL, "d@g.net").build();

Note, that all of the features are statically typed and don’t use reflection.

What do you think?

“A quoi sert un serveur d’intégration continue ?” C’est la question que nous posons chez Tech4Quant. Des années à prêcher pour que chaque équipe ait un serveur d’IC. Nous avions installé et utilisé Hudson dès le début du projet. Et voilà que depuis plusieurs mois, Hudson installé on ne sait plus où, échoue à chaque commit et personne ne s’en émeut.

Comment en arriver là ?

Le déclencheur a été le temps de maintenance du serveur Hudson. Il doit avoir en permanence toutes les briques nécessaires au build. Notre produit nécessite malheureusement que latex, R et mono soient installés, ainsi que plusieurs packages R. Il faut mettre à jour le serveur régulièrement, le rebooter parfois, faire du ménage à l’occasion.

Pourquoi passer du temps à maintenir un serveur lorsque nous pourrions nous assurer de passer tous les tests en local avant de pousser le code sur le repository partagé ?

Décision prise: petite équipe + rigueur + build rapide = Adieu Hudson!

Un build de plus en plus lent

Seulement voilà, tout build devient forcément de plus en plus long. Une minute, puis deux, puis cinq. Passer tous les tests en local devient un obstacle. Il est tentant de pousser vers un serveur d’IC sans passer les tests fonctionnels et d’attendre le verdict, un café à la main. Résultat : soit cinq minutes devant maven, soit un paquet de builds cassés.

Course aux armements

Devait-on mettre en place un mécanisme de commit à deux phases avec un serveur sas ? Devait-on remplacer Hudson par TeamCity qui propose aux utilisateurs d’Idea un private build distant, pré-requis au commit définitif ? Cela sonne “Course aux armements” dans ma tête. Bientôt nous aurions une ferme de build à maintenir. Vision d’horreur !

Réduire le temps du build

Beaucoup plus productif que d’investir dans de nouveaux serveurs, attaquons nous à la cause racine plutôt qu’à un symptôme. Tentons de réduire le temps pris par les tests. Après un peu de magie noire (un futur post), voilà notre build complet passé de plus de cinq minutes à moins de deux. On respire à nouveau. Mais la question initiale reste en suspend. Comment repousser le moment fatidique où il faudra investir dans un serveur d’IC pour absorber le temps de build invariablement croissant ?

Git, le sauveur

Nous utilisons git. Sa stabilité et la simplicité d’installation sont de forts atouts. L’utilisation au jour le jour est, elle, un peu ardue étant donné la médiocrité des front-ends.

En relisant “Git Magic”, je me dis qu’il serait simplissime de mettre au point un build privé quasi-identique à celui de TeamCity, sans serveur distant. Dupliquons le code source de notre répertoire de travail vers un second répertoire depuis lequel on lance le build et en cas de succès on fait un commit.

Avec une autre gestion de conf, deux problème se poseraient : combien de temps prendra la duplication du projet lorsqu’il deviendra gros ? Un fois les résultats poussés depuis le répertoire dupliqué, le répertoire principal n’est pas forcément “au courant” que toutes ses modifications ont été poussées par quelqu’un d’autre.

Avec git, rien de tout ça. La première fois que l’on duplique le répertoire de travail avec ‘git clone’, git copiera tous les fichers de manière ultra-rapide (lien). Les fois suivantes, git saura mettre à jour la copie par simple delta en un claquement de doigts. Impressionnant.

Pour ce qui est de la resynchronisation du répertoire de travail, un simple ‘git pull’ se rendra compte, grâce au hash des fichiers delta, que le répertoire courant contient exactement les même fichiers et les mêmes commits que ceux qui ont été poussés sur le repository partagé. Donc rien à faire. Re-impressionnant.

Le build privé ultime ?

Et voilà un build privé sans maintenance, sans installation supplémentaire, indépendant de l’environnement de travail, éxécuté en une seule commande. Plus un seul build n’est en erreur dans le répository partagé et vous pouvez mettre à la poubelle votre serveur d’IC.

Car c’est bien une Intégration Continue sans serveur que l’on vient de mettre en place. Tout ce qu’un serveur d’IC sait faire en plus est à mon avis du bruit de fond.

La recette

Pour les plus curieux, voici le script:

#!/bin/bash
if [ 0 -eq `git remote -v | grep -c push` ]; then
	REMOTE_REPO=`git remote -v | sed 's/origin//'`
else
	REMOTE_REPO=`git remote -v | grep "(push)" | sed 's/origin//' | sed 's/(push)//'`
fi

if [ ! -z "$1" ]; then
	git add .
	git commit -a -m "$1"
fi

git pull

if [ ! -d ".privatebuild" ]; then
	git clone . .privatebuild
fi

cd .privatebuild
git clean -df
git pull

if [ -e "pom.xml" ]; then
	mvn clean install

	if [ $? -eq 0 ]; then
		echo "Publishing to: $REMOTE_REPO"
		git push $REMOTE_REPO master
	else
		echo "Unable to build"
		exit $?
	fi
fi

Hi all,

This blog was moved to http://javabien.net. Please update your bookmarks or better add this blog to your favorite news reader.

David.