Archive for August, 2010

iPad market share

John Gruber commenting on an article predicting a sharp drop in iPad market shares:

And the evidence of platforms winning solely on the basis of “openness” is what? And the Android handset maker selling more units or making more profit than Apple (or RIM) is who?

Yeah, yeah. We get it, John. Apple is #1 in profits and #1 in manufacturers selling handsets.

Which is great for Apple.

And terrible for consumer choice.

Atari 2600 Box Art

I remember a lot of these boxes and curiously, the deceptive advertising never bothered me. Back then, you just had to make up the absence of graphics with your imagination.

Still, funny by today’s standards.

The best way to learn French

Easy: Buy this book.

It’s only $10.

“Wait a minute”, you say… “This is not even a language learning book, it looks like a comic strip and it’s not even in French, how would it be the best way to learn French?”.

First of all, this is one of the best stories I have read in a book, novel or comic. The scenario starts very conventionally and slowly drives you into directions that will keep you guessing until the very end.

Consider this book as an introduction to “Bandes Dessinées”, a very important part of French culture that children and adults enjoy throughout their life. “Bande dessinées” (usually referred to as “BD” in France) literally means “drawn strip” but “comic strip” is close enough. However, it’s very different from the comic strips that you are used to if you live in the US.

Not only are the BD’s a very fertile ground for countless artists but writers sometimes dedicate their entire career to BD’s over novels. That’s how strong the phenomenon is in France.

So what makes this book so special?

First of all, the story is from Jean Van Hamme, one of the most inventive story writers that I’ve ever had the pleasure to read. To me, he sits right there with the likes of Alastair Reynolds, Peter Hamilton, J.K. Rowling, Robert Ludlum and Dan Simmons. You will find him as the prolific author of many other series, but I find this one to be really his masterpiece.

“The Master of the Mountains” is part of the Thorgal series. It’s book number fifteen in the series, but since very few of them were translated in English, it has a different number in English. Fortunately, you don’t need to have read the previous books to enjoy this particular book, but it’s an exception more than a rule: Van Hamme is the master of plots that span over dozens of books and not only does Thorgal deliver compelling and imaginative stories, it also hinges over a common story arc that is slowly revealed book after book. This is not your standard viking war story.

I’m going to resist the urge of dropping even the slightest hint about what “The Master of the Mountain” is about. The less you know about it when you turn to the first page, the more you’ll enjoy it.

Once you’re done with this book, I am betting that your appetite will be whetted and you will want to know more about Thorgal, French bandes dessinées and, hopefully, French language itself. Maybe you will be tempted to buy the French version of this album, even though it’s a bit more expensive ($25) and start learning French by comparing the two side by side.

Once you are done with Thorgal (there are 31 books so far, the next one will be out in November), you will probably want to start looking at the XIII series, that Van Hamme authored as well, and which tells the tale of someone who wakes up one day having forgotten everything about his past and trying to piece it together with the only piece of information that he has: the number “XIII” tatooed on his neck (if this sounds familiar to you, it does use the same premise as the Bourne series, but it quickly goes into very different territory).

Clojure, concurrency and silver bullets

Rob Martin has become a fan of Clojure recently. Nothing wrong with that, Clojure has a lot going for it and if you’ve never had a chance to write code in Lisp, it’s probably the best way to begin these days.

But then, Rob gets a little bit too emotional and he starts drawing all kinds of dangerous conclusions. Such as this one:

Why is functional programming important? Because Moore’s law has started to falter.

It’s not the first time that functional programming gets advocated as the heroic technology that will rescue us from buggy multithreaded code *and* that it will allow our programs to magically scale along with the multiple cores that computers have these days. Concurrency problems? Just pick a functional programming language — any language — and suddenly, your code is multithread safe and it will automatically scale.

I find this simplification a bit disappointing coming from technologists, but I really read this at least once a week these days.

If you’ve ever written multi-threaded code, the thought of eight, sixteen, thirty-two, or even more processors running your program should fill you with dread. Writing multi-threaded code correctly is hard! But why is it so hard? Because it is hard to manage the state of variables when more than one CPU has access to them.

First, a nit: when you write multi-threaded code, four processors shouldn’t scare you more than two. Either your code is multi-threaded safe or it’s not. The only thing that changes when you run it on multiple processors is that you are more likely to find bugs when you throw more processors at it.

I’ll agree with Rob on the fact that managing the state of variables with more than one CPU is hard, but come on, it’s still not rocket science. As I write this, hundreds of thousands of lines written in C, C++, C#, Java and who knows what other non functional programming languages are running concurrently, and they are doing just fine.

Java has shown amazing powers of adaptation over the years, and when it comes to concurrency, people like Brian Goetz and Doug Lea and libraries such as java.util.concurrent don’t get the recognition they deserve. Java is also the living proof that you don’t need concurrency support at the language level to be effective, libraries can do just fine.

That kind of code is admittedly harder to write than straight imperative programming, but can anyone who’s looked at Clojure’s STM API (atoms, agents, ref) or Scala and Erlang’s actors say that writing code with these paradigms is that much easier?

To make matter worse, these new paradigms come at a cost that’s very often glossed over by their own advocates. When people tell you that “Actors are a share nothing architecture”, they are lying to you. You are sharing a great deal with actors, just in more subtle ways that your mind needs to be very aware of. You have the illusion of automatic multi thread safety but you pay the price by having to wrap your head around a fully asynchronous model. It’s not easy. And when you have two actors A and B that are sending messages to an actor C, aren’t they sharing the state of C?

Stephan Schmidt attacked this subject not long ago. Read his post and don’t miss the comments, they are very enlightening as well. My take away from that discussion is that if there is a silver bullet to concurrency programming, Actors are not it.

Actually, it’s pretty clear to me that there is no silver bullet, and Alex Payne seems to agree. In this post, Alex sends a very powerful message of compromise and inclusion. Blocking or non-blocking I/O? Select or events? Java locking or Actors? Agents or refs?

Anyone who tells you that only one of these approaches works and the others don’t is trying to sell you something.

To quote Alex:

In fact, taking a hybrid approach to concurrency seems to be the way forward if the academy is any indication.

Strive to learn new languages, technologies, paradigms, just don’t fall in love with them.

Fun with the Anthropic Principle

I have been fascinated by the Anthropic principle for a very long time. It also took me a while to understand it, and while I can’t claim to grasp its implications fully, I have reached a point where I see this principle around me all the time and it’s giving me a warm feeling of appreciation for the world that surrounds me.

The Anthropic Principle is also the ultimate weapon against the obscurantism pushed forth by proponents of creationism or intelligent design.

I’ve been thinking of an easy way to explain the Anthropic Principle so I thought I’d start by offering it here. I’m not quite sure where this story came from, if I read it somewhere (probably) and even if somebody ever connected it to the Anthropic Principle, so feel free to comment at will.

Story time.

One day, someone called Steve sends you an email in which he predicts that tomorrow, team A will win against team B. You don’t think much of that email and you delete it. The next day, you learn that indeed, team A won. A few days later, you receive another email from Steve which, again, makes a prediction for the result of an upcoming game. And again, the prediction turns out to be correct.

After a while, you have received ten emails from Steve, each of which accurately predicted a game outcome. You start being quite shocked and excited. What are the odds that this person would randomly guess correctly ten matches? 1 over 2^10 (1024), about 0.1%. That’s quite remarkable.

In his next email, Steve says “I hope that by now, I convinced you that I can guess the future. Here is the deal: send me $10,000, I’ll bet them on the next match and we’ll split the profits”.

Do you send the money?

Surprisingly, a lot of people fall for this kind of scam on a daily basis. If you think about this a little bit, you can probably see how the scammer did it: he started by sending a prediction that A will win to 512 recipients and one where B will win to the other 512. After the game is finished, he repeats the process with the 512 that received the right result. Every time a new match result comes in, the number of recipients is divided by two, but the remaining recipients have all received 100% accurate predictions so far.

By the time we reach the 10th match, there is only one recipient left — you. And to you, the sender of this email has proven that he has an uncanny ability to guess the future while all he did is walk through an entire solution space until he had reached a point where he can carry out his scam.

How does this short story relate to the Anthropic Principle?

Proponents of creationism and intelligent design usually make claims along the lines of observing the amazing complexity that lies around us and ascribe these observations to the existence of a god. This is very similar to yourself learning about this person who seems to be able to guess the future. Surely, only the existence of a supreme being can explain for such amazing feats, right?

Here are other similar claims:

  • “The eye is such a complex organ that it couldn’t have evolved to become what it is, it must have been created by someone”.
  • “If the atmosphere mix of the Earth had been off by just a few percents, human life would not be possible”.
  • “Our cosmos would not exist if the constants that underlie it were off by just a tiny fraction of a decimal”.

Are all these numbers so remarkable that the only way to explain them is by the existence of a supreme being?

Of course not. There are millions of universes that are similar to ours, and which have all these microscopic variations in their constants. And if you’re not convinced, it’s easy: just go ask the people who live in these universes. Except that… you can’t, of course. Because life never emerged in these universes.

What happened is that you got lucky: you were born in one of these few universes where life was possible. Unlucky people never realized that they were unlucky since they were never born, and as such, they were never able to ponder these questions.

This realization is the very definition of the anthropic principle: all these magic values that surround us and that make life possible are actually unremarkable, because they, and you, are the product of a statistical event. There is nothing so magical about our eyes that can only be explained by the existence of a supreme being. The simple truth is that if the eye was not the complex organ that it is today, you wouldn’t be around to ask questions about it.

Interestingly, the Anthropic Principle is not mutually exclusive with the existence of a deity (actually, nothing really is, which is part of the problem). You can still believe that some god decided that you would be part of the lucky experiment, but the Anthropic Principle is certainly very strong supporting evidence for the mechanism of evolution and also the proof that a lot of the seemingly magical properties that permeate the world around us can be very simply explained by high school level probability concepts.

YAML, the forgotten victim of the format wars

I have been thinking about providing an additional way of specifying test suites for TestNG for a while. The current format is XML, which has quite a few benefits, but no matter how hard you try to specify an XML file to make it as concise as possible, the end result will always end up being more verbose than you’d really like, especially if that format grows organically throughout the years and ends up having to cover cases that you didn’t envision initially. I designed the TestNG DTD in 2004, and while I think it’s evolved reasonably well, I’m looking for a format that might be easier to work with.

YAML attracted my attention when I started playing with Ruby on Rails a few years ago. It’s fairly expressive and concise, but for some reason, it never really took off in the Java world. These past few days, I started experimenting with YAML and TestNG, and I have to say that the initial results are pretty promising.

Here is a simple testng.xml:


 


  <parameter name="n" value="42" />

  
    
      
        <exclude name="broken" />
      
    

    
      <class name="test.listeners.ResultEndMillisTest" />
    
  

and here is its YAML version:

name: SingleSuite
threadCount: 4
parameters: { n: 42 }

tests:
  - name: Regression2
    parameters: { count: 10 }
    excludedGroups: [ broken ]
    classes:
      - test.listeners.ResultEndMillisTest

The result is even more telling on bigger files:

-rw-r--r--  1 cbeust  502  17638 Aug  6 11:07 src/test/resources/testng.xml
-rw-r--r--  1 cbeust  502  10732 Aug 12 16:03 src/test/resources/testng.yaml

That’s almost a 40% reduction in size. And the gain in readability is also pretty obvious: here is testng.xml and testng.yaml. By the way, this is the main file that describes all the TestNG tests and the YAML file is completely equivalent to its XML version.

Adding a YAML front-end to TestNG turned out to be pretty easy. I found several libraries and I ended up picking SnakeYAML because it still seemed relatively active (the last activity dates from January) and also because it’s available in Maven.

TestNG’s XML front-end is entirely captured in the org.testng.xml package, which only contains six classes. Each of these classes maps exactly to a TestNG tag (<suite> to XmlSuite, <test> to XmlTest, etc…). As long as you can hand an XmlSuite object that defines the root of your suite to TestNG, the engine doesn’t care what file was used to produce it.

The only task was therefore to parse the YAML file and create all the XML objects that TestNG expects. YAML makes this pretty easy since it lets you map keywords to classes.

The entire effort is contained in these few lines:

Constructor constructor = new Constructor(XmlSuite.class);

TypeDescription suiteDescription = new TypeDescription(XmlSuite.class);
suiteDescription.putListPropertyType("packages", XmlPackage.class);
suiteDescription.putListPropertyType("listeners", String.class);
suiteDescription.putListPropertyType("tests", XmlTest.class);
suiteDescription.putListPropertyType("method-selectors",
    XmlMethodSelector.class);
constructor.addTypeDescription(suiteDescription);

TypeDescription testDescription = new TypeDescription(XmlTest.class);
testDescription.putListPropertyType("classes", XmlClass.class);
testDescription.putMapPropertyType("metaGroups", String.class, List.class);
testDescription.putListPropertyType("method-selectors",
    XmlMethodSelector.class);
constructor.addTypeDescription(testDescription);

Loader loader = new Loader(constructor);
org.yaml.snakeyaml.Yaml y = new org.yaml.snakeyaml.Yaml(loader);
FileInputStream is = new FileInputStream(new File(filePath));
XmlSuite result = (XmlSuite) y.load(is);

This part was really easy. I had to make a few additional adjustments that I’ll gloss over, but overall, the process was very smooth.

Additionally, since XML objects can dump themselves in XML, adding the similar functionality for YAML gave me an XML <-> YAML converter for free, which will come in handy for users who want to convert their files or just see for themselves how their XML files will look once converted to YAML.

As for YAML itself, the specification makes it look harder than it really is, so here are a few simple rules:

  • You can define key-value pairs, lists and maps.
  • Lists and maps can be specified either on one line or on several lines.
  • Single line list: [ a, b, c ]
  • Single line map: { a:1, b:2, c:3 }
  • Multi line list:
        - a
        - b
        - c
      
  • Multi line map:
        a: 1
        b: 2
        c: 3
      

There are a couple of downsides to YAML: there are not a lot of tools available for it and it’s also not very easy to validate (I am not aware of anything similar to a DTD or a schema). Despite these limitations, I’m still very tempted to officially add support for YAML in TestNG because of the convenience it brings.

If you have any experience to share about YAML, please feel free to comment.

Patent irony

Interestingly, two of the patents that Oracle is suing Google over were filed by now Google employees:

I don’t mean to single out these two individuals, this is business as usual. Tech companies are pretty good at providing financial incentives to get their employees to file patents, and as long as these patents pass the simple “prior art” Litmus test, the company will be happy to file them and add them to their portfolio. I have filed my share of patents at the companies I worked on, including Sun (that was more than ten years ago).

Oracle is suing Google over Java. There’s just one tiny problem…

Oracle is suing Google over Java.

I’m really curious to see what legal ground Oracle is going to be using considering that Sun’s previous CEO, Jonathan Schwartz, officially congratulated Google the day they announced Android:

I just wanted to add my voice to the chorus of others from Sun in offering my heartfelt congratulations to Google on the announcement of their new Java/Linux phone platform, Android. Congratulations!

Cached link, in case it gets removed.

This has always struck me as a very odd thing to do for a CEO considering that Android didn’t license Java (and as such, was not allowed to use the word “Java” anywhere). I’m guessing that such a blog post would count as a clear indication that Sun was not interested in aggressively defending the Java trademark, and once you go down that path, you lose a lot of credibility in court.

I think it’s going to be an uphill battle for Oracle, but I have to give credit to Larry Ellison for making the tough choices that the two past Sun CEO’s never had the courage to make.

Complex line command syntaxes with JCommander

Complex tools such as git or svn understand a whole set of commands, each of which with their own specific syntax:

  git commit --amend -m "Bug fix"

Words such as “commit” above are called “commands” in JCommander, and you can specify them by creating one arg object per command.

For example, here is the arg object for the commit command:

@Parameters(separators = "=")
public class CommandCommit {

  @Parameter(description = "Record changes to the repository")
  public List<String> files;

  @Parameter(names = "--amend", description = "Amend")
  public Boolean amend = false;

  @Parameter(names = "--author")
  public String author;
}

And here is add:

public class CommandAdd {

  @Parameter(description = "Add file contents to the index")
  public List<String> patterns;

  @Parameter(names = "-i")
  public Boolean interactive = false;
}

Then you register these commands with your JCommander object. After the parsing phase, you call getParsedCommand() on your JCommander object, and based on the command that is returned, you know which arg object to inspect (you can still use a main arg object if you want to support options before the first command appears on the command line):

// Options available before commands
CommandMain cm = new CommandMain();
JCommander jc = new JCommander(cm);

// Command: add
CommandAdd add = new CommandAdd();
jc.addCommand("add", add);

// Command: commit
CommandCommit commit = new CommandCommit();
jc.addCommand("commit", commit);

jc.parse("-v", "commit", "--amend", "--author=cbeust", "A.java", "B.java");

Assert.assertTrue(cm.verbose);
Assert.assertEquals(jc.getParsedCommand(), "commit");
Assert.assertTrue(commit.amend);
Assert.assertEquals(commit.author, "cbeust");
Assert.assertEquals(commit.files, Arrays.asList("A.java", "B.java"));

Support for complex commands is one of the new features available in JCommander 1.5.

Announcing TestNG 5.13.1

TestNG 5.13.1 is now available. The previous release was just three months ago, but as you can see below, a lot of fixes went into this new version. A big thanks to everyone who helped (hopefully credited below), and especially to Nalin Makar, who made some major contributions to TestNG these past months.

You can either download TestNG from the web site or from Maven:

    
      org.testng
      testng
      5.13.1
      test
    

Here is the complete list of changes, please refer to the documentation for more details.

TestNG

  • Added: -methods
  • Added: -configfailurepolicy (Todd Quessenberry)
  • Added: -methodselectors (Todd Quessenberry)
  • Added: @NoInjection
  • Added: <test preserve-order="true">
  • Added: -testnames (command line) and testnames (ant)
  • Added: New ant task tag: propertyset (Todd Wells)
  • Added: ITestNGListenerFactory
  • Added: Passing command line properties via the ant task and doc update (Todd Wells)
  • Added: Hierarchical XmlSuites (Nalin Makar)
  • Added: Reporter#clear()
  • Fixed: NullPointerException when a suite produces no results (Cefn Hoile)
  • Fixed: Identical configuration methods were not always invoked in the correct order in superclasses (Nalin Makar)
  • Fixed: @DataProvider(parallel = true) was passing incorrect parameters with injection
  • Fixed: Replaced @Test(sequential) with @Test(singleThreaded)
  • Fixed: If inherited configuration methods had defined deps, they could be invoked in incorrect order (Nalin Makar)
  • Fixed: Initialize all Suite/Test runners at beginning to catch configuration issues right at start (Nalin Makar)
  • Fixed: Issue7: Issue86 Incorrect dates reported for configuration methods
  • Fixed: Issue24: OOM errors in SuiteHTMLReporter (Nalin Makar)
  • Fixed: Time outs specified in XML were not honored for <suite parallel=”tests”>
  • Fixed: <suite> and <test> time outs were hardcoded, they now honor their time-out attribute
  • Fixed: TestNG was hanging if no test methods were found
  • Fixed: onTestSuccess() was called after @AfterMethod instead of after the test method (test: test.listener.ListenerTest)
  • Fixed: XML test results contained skipfailedinvocationCounts instead of skipfailedinvocationcounts
  • Fixed: Issue4 assertEquals for primitive arrays, Issue34 assertNull javadoc updated
  • Fixed: Issue78 NPE with non-public class. Now throws TestNG exception
  • Fixed: NPE with @Optional null parameters (Yves Dessertine)
  • Fixed: TESTNG-387 TestNG not rerunning test method with the right data set from Data Provider (François Reynaud)
  • Fixed: Show correct number of pass/failed numbers for tests using @DataProvider
  • Fixed: Return correct method status and exception (if any) in InvokedMethodListener.afterInvocation()
  • Fixed: Trivial fixes: TESTNG-241 (log message at Info), Issue2 (throw SAXException and not NPE for invalid testng xml)
  • Fixed: Configuration methods couldn’t depend on an abstract method (Nalin Makar)
  • Fixed: TestNG#setTestClasses was not resetting m_suites
  • Fixed: Exceptions thrown by IInvokedMethodListeners were not caught (Nalin Makar)
  • Fixed: @Listeners now works on base classes as well
  • Fixed: Test priorities were not working properly in non-parallel mode
  • Fixed: @Listeners wasn’t working properly with ITestListener

Eclipse

  • Fixed: TESTNG-395 New wizard was creating classes called “NewTest”
  • Fixed: TESTNG-397 Class level @Test was preventing groups from showing up in the launch configuration

Doc

  • Updated Maven documentation (Brett Porter)