Otaku, Cedric's blog

Dark Souls

Cedric — Thu, 26 Apr 2012 18:06:15 +0000

I heard about Dark Souls when it came out because it was said to be incredibly difficult. I made a mental note of trying it out one day and then forgot about it. A few weeks ago, two good friends mentioned it to me again with high praises, so I decided to give it a shot. I dusted off my Playstation 3 (I’m pretty much exclusively a PC gamer), grabbed the game ($35 on Amazon, quite a bargain) and launched it with an open mind.

Nothing could have prepared me for what happened next.

Be prepared to die. And die again. And again. And again.

While the game starts with the standard tutorial on how to use the controller to move your character about and how to fight, I wasn’t exactly ready to face a boss five minutes into the tutorial and immediately die. Then try again, and die. And die, again and again. Ever time I fight it, I chip at his health a bit more but I realize that my sword hilt is barely damaging it at all, and at this pace, it would take me a very, very long time to kill it, assuming I manage to stay alive for that long.

I don’t want to spoil the game so I’ll just say that there is a way out, obviously, and once you figure it out, you realize that the game wasn’t being “stupidly” hard, it was just being “hard but fair”. It was giving you a small taste of what lies ahead. And that looked not just deliciously refreshing but quite enticing as well. You will die a lot in Dark Souls, but most of the time, it will be your fault and you will learn from it. And the next time you try, you’ll get a bit further.

Having said that, limiting the description of this game to this simple observation doesn’t come even remotely close to doing it justice.

Much, much deeper than it looks.

First of all, Dark Souls has the best combat system I have ever seen in more than thirty years of gaming. It’s not even close. Not only do you need to learn to hit, but also to dodge, to parry, to roll, to position yourself correctly, to backstab and to riposte. You also need to learn your opponent. Each of them (trash and boss) have cues that tell you what the next attack is going to be, and each of these attacks can be mitigated or completely avoided if you know what to do.

A bonfire, your best friend in the game.

Next, you have the enemy population. I’ve been constantly impressed with the level of attention that the developers have put in making the game challenging but manageable. There is only one level of difficulty on Dark Souls (with a small caveat, see below), so you don’t get to choose how hard or how numerous the enemies are. But they are planted along your way in very, very calculated ways that will make you want to be extremely mindful when you progress through new areas. This is not the game where you rush in a pack of enemies and mash your controller buttons until you are surrounded with carcasses. As a matter of fact, dealing with more than one enemy at a time will very often result in a quick death.

The leveling is also quite innovative. As opposed to most RPG/MMORPG’s, you are in complete control of how your character progresses. When you kill enemies, you earn “souls” which you can then use either to level, buy things or upgrade your existing gear. There is never a better path on what to do with all these souls, and the game is so well tuned that you can put all your souls into equipment upgrades and complete it at level 1.

When you die, all your souls drop on the ground and you need to run back there to retrieve them. However, if you die on your way to retrieve them, these souls are lost. This is a very simple mechanism (halfway between World of Warcraft and Everquest, which was very punishing) but one that creates a very palpable and constant tension. You don’t care too much about dying when you’re walking around with a small number of souls, but as you progress, that number grows and very soon, you start wondering if pressing ahead is not too risky and if you shouldn’t turn around, go back to a check point (bonfire) and spend these souls, which you can’t bank. Obviously, the fact that in new areas, the mobs are strategically positioned to kill you if you’re not careful makes for some very tense moments (“do not die, do not die!”). It’s not very often that playing a game raises my heart rate and hand clamminess, but Dark Souls definitely has this effect on me on a regular basis.

Who’s the boss?

The bosses are probably the least innovative aspect of Dark Souls. Don’t get me wrong, it’s nowhere near “Deus Ex Human Revolution bosses” bad: the Dark Souls bosses are still fairly challenging to the point where initial encounters usually result in a oneshot death within thirty seconds, but so far, I haven’t seen incredibly innovative mechanics. The bosses are still a lot of fun to figure out and I strongly recommend not reading any strategy beforehand, because killing bosses after figuring them out all by yourself alwas makes for an amazing feeling of elation. I do think the value of Dark Souls is more that the attention to combat details applies to all the mobs and not just the bosses.

The Hydra, one of the many, many bosses you will encounter.

Here is another innovative aspect of Dark Souls: when you reach a bonfire, all the enemies respawn (except bosses and a few special enemies). I was quite baffled when I discovered this, but I’m now realizing how important this aspect is to the Dark Souls atmosphere and philosophy. I won’t dig too deep in the details, but this aspect opens up the possibility of farming, which gives you an always available option to improve your character in case you think it’s not strong enough for the area you’re trying to explore.

Tools of the trade

Do you enjoy gearing up and tuning your weapons and armor? Dark Souls has plenty to offer in that area as well, with a mindbogglingly number of items to find and a lot of varied ways to enchant them the way you want them. Like many things, there is not a single true path there and it’s much more important to use and tune a weapon that fits your play style than picking up the one with the highest numbers.

Dark Souls is also very open in terms of character evolution. When the game starts, you get to choose the class that you want, but this class is not much more than a set of stats that are precalculated for you. When comes the time to level, you are free to put these souls into whatever stat you feel like, and if you started with a Knight but decided that you wanted it to cast a decent number of spells or wield daggers and move around very fast, just allocate the points in the correct categories (Faith and Dexterity) and you’ll create the character that represents your style of play. This flexibility of character build is what is prompting players to restart the game many times over and experiment with various builds. However, many choose to continue on with the same character, which is another intriguing aspect of Dark Souls.

Here is how it works.

Beyond the first play through

I haven’t completed the game yet, so I’m only repeating what I’ve read, but once you beat the final boss, you can reset the entire game and start from scratch. The new game (referred to as “NG+”: New Game Plus) features tougher enemies and bosses, and obviously, better rewards in gear and souls. Quite a few people seem to enjoy this aspect of the game as well since I’ve seen many say that they were on their third, fifth or tenth (NG+10!) play through.

Two is a crowd

One final noteworthy and innovative aspect of Dark Souls is multiplayer. First of all, you can play the entire game offline without any problems, but you will miss out on some very neat features of the game. If you choose to play while signed in, a few interesting multiplayer options open up.

On top of the traditional duels against human opponents, you can invade someone else’s game or be invaded. These invasions can be either co-operative or hostile.

For co-op play, you can leave a mark letting everyone know that you are willing to be summoned. If a player chooses to summon you, you will materialize in their world and, most likely, help them out with a boss that they’re having trouble with. Being summoned this way gives you a chance to observe that boss more carefully so you are more prepared to kill it on your own. Alternatively, you can ask for help, which can come either in the form of another player or from certain NPC’s, which are made available throughout the game for specific bosses.

Invasions can also be hostile, but you need to explicitly make yourself available to them in order for PvP to occur (like flagging yourself at WoW). In Dark Souls, you do that by “regaining” humanity. By default, you are not human (the game calls this state “hollow”). Regaining humanity will net you added rewards but it will also make you available for PvP, which means that other players can then invade you. When this happens, the current area becomes sealed and the PvP phase will only end when one of the two players dies (or the phantom withdraws). I’m not much of a PvP person myself, but I have to say that receiving the suddent notice that “Player xxx has invaded!” on your screen is always a bit stressful, and you start getting paranoid trying to locate that person before they find you.

In red, a phantom that has invaded your world.

A couple more interesting aspects of the online play: 1) players can leave messages for everyone else to see (“Chest ahead”, “Be wary of left”, etc…) and 2) you can see where and how other online players died by touching their bloodstain. In theory, this is supposed to give you an idea of the kind of trap or trash pull that lies ahead, but in practice, I can’t say I’ve found much value in bloodstains.

To the top with you!

I have played more than one hundred hours so far. I’m probably about 2/3rd through the game, and I’m still loving every minute of it. In more than thirty years of video games, I can only think of two games that I ever played more than one hundred hours: World of Warcraft and Civilization. Dark Souls has definitely earned the title “One of the best games I have ever played” with its mix of innovative features, amazing tuning, clever mechanics and engrossing world. Both WoW and Civilization have given me incredible amounts of satisfaction and fun, but Dark Souls goes beyond this by making me stressed, tense and incredibly elated when I finally overcome a difficult part.

It’s currently available on Playstation 3 and XBox 360, but a PC version is in the works and scheduled to ship this summer.

You’ve been implementing main() wrong all this time

Cedric — Mon, 26 Mar 2012 01:36:45 +0000

Since the very early days of Java (and C-like languages overall), the canonical way to start your program has been something like this:

public class A {
  public static void main(String[] args) {
    new A().run(args);
  }

  public void run(String[] args) {
    // Your application starts here
  }
}

If you are still doing this, I’m here to tell you it’s time to stop.

Letting go of ‘new’

First, install Guice in your project:

        
          com.google.inject
           guice
           3.0

and then, modify your main method as follows:

public class A {
  public static void main(String[] args) {
    Injector.getInstance(A.class).run(args);
  }
}

So, what does this buy you exactly?

You will find a lot of articles explaining the various benefits of Guice, such as being able to substitute different environments on the fly, but I’m going to use a different angle in this article.

Let’s start by assuming the existence of a Config class that contains various configuration parameters. I’ll just hardcode them for now and use fields to make the class smaller:

public class Config {
  String host = "com.example.com";
  int port = 1234;
}

This class is a singleton, it is instantiated somewhere in your main class and not used anywhere else at the moment. One day, you realize you need this instance in another class which happens to be deep in your runtime hierarchy, which we will call Deep. For example, if you put a break point in the method where you need this config object, your debugger would show you stack frames similar to this:

com.example.A.main()
com.example.B.f(int, String)
com.example.C.g(String)
com.example.Deep.h(Foo, int)

The easy and wrong way to solve this problem is to make the Config instance static on some class (probably A) and access it directly from Deep. I’m hoping I don’t need to explain why this is a bad idea: not only do you want to avoid using statics, but you also want to make sure that each object is exposed only to objects that need them, and making the Config object static would make your instance visible to your entire code base. Not a good thing.

The second thought is to pass the object down the stack, so you modify all the signatures as follows:

com.example.A.main()
com.example.B.f(int, String, Config)
com.example.C.g(String, Config)
com.example.Deep.h(Foo, int, Config)

This is a bit better since you have severely restricted the exposure of the Config object, but note that you are still making it available to more methods than really need to: B#f and C#g have really nothing to do with this object, and a little sting of discomfort hits you when you start writing the Javadoc:

public class C {
  ...
  /**
   * @param config This method doesn't really use this parameter,
   * it just passes it down so Deep#h can use it.
   */
  public void g(String s, Config config) {

Unnecessary exposure is actually not the worst part of this approach, the problem is that it changes all these signatures along the way, which is certainly undesirable in a private API and absolutely devastating in a public API. And of course, it’s absolutely not scalable: if you keep adding a parameter to your method whenever you need access to a certain object, you will soon be dealing with methods that take ten parameters, most of which they just pass down the chain.

Here is how we solve this problem with dependency injection (performed by Guice in this example, but this is applicable to any library that implements JSR 330, obviously):

public class Deep {
  @Inject
  private Config config;

and we’re done. That’s it. You don’t need to modify the Config class in any way, nor do you need to make any change in any of the classes that separate Deep from your main class. With this, you have also minimized the exposure of the Config object to just the class that needs it.

Injecting right

There are various ways you can inject object into your class but I’ll just mention the two that, I think, are the most important. I just showed “field injection” in the previous paragraph, but be aware that you can also prefer to use “constructor injection”:

public class Deep {
  private final Config config;

  @Inject
  public Deep(Config config) {
    this.config = config;
  }

This time, you are adding a parameter to the constructor of your Deep class (which shouldn’t worry you too much since you will never invoke it directly, Guice will) and you assign the parameter to the field in the constructor. The benefit is that you can declare your field final. The downside, obviously, is that this approach is much more verbose.

Personally, I see little point in final fields since I have hardly ever encountered a bug that was due to accidentally reassigning a field, so I tend to use field injection whenever I can.

Taking it to the next level

Obviously, the kind of configuration object I used as an example if not very realistic. Typically, a configuration will not hardcode values like I did and will, instead, read them from some external source. Similarly, you will want to inject objects that can’t necessarily be instantiated so early in the lifecycle of your application, such as servlet contexts, database connections, or implementations of your own interfaces.

This topic itself would probably cover several chapters of a book dedicated to dependency injection, so I’ll just summarize it: not all objects can be injected this way, and one benefit of using a dependency injection framework in your code is that it will force you to think about what life cycle category your objects belong to. Having said that, if you want to find out how Guice can inject objects that get created at a later time in your application life cycle, look up the Javadoc for the Provider class.

Wrapping up

I hope this quick introduction to dependency injection piqued your interest and that you will consider using it in your project since it has so much more to offer than what I described in this post. If you want to learn more, I suggest starting with the excellent Guice documentation.

Video game update

Cedric — Sat, 10 Mar 2012 21:10:40 +0000

I haven’t talked about video games in a while, so I thought I’d put together a short summary of what I played this past year, in no particular order.

Deus EX Human Revolution

The original Deus Ex is considered to be one of the best PC games of all times (along with another one mentioned below), so developing a sequel more than ten years later is quite a tall order. From what I can see, DXHR manages to pull it off. I played about one third through the game and then put it away with the firm intention to go back to it (a pattern that seems to be recurring in my gaming habits).

Portal 2

I don’t think Portal needs any introduction, and nobody actually expected a sequel to the best PC game of all times (according to PC Gamer) to be able to measure up. This is why Portal 2 was such a delightful surprise to me, and in hindsight, I think it’s better than the first episode in all respects. On top of inventing new creative puzzles, Valve managed to actually imagine a very poignant and appealing story that I absolutely fell in love with. I replayed the game three times and I suspect that I will do so again on a regular basis in the years to come. The characters, their acting and voices are also incredibly well done and downright funny despite the tense scenario, and I don’t think anyone will be able to forget the Stephen Merchant / Ellen Mc Lain duo for a long time.

Civilization V

I have always been a Civilization fan, going back to the very first installment of the series and its insanely punishing difficulty. Civilization V doesn’t betray the series and streamline a lot of the rough edges that players complained about in the previous version. List most Civilization, I have the feeling I barely scratched the surface of this one and I’m looking forward to going back to explore it further. In my book, however, Civilization IV remains the best of the entire series.

Terraria

Referring to this little gem as “Minecraft in 2D” is not doing it justice. It shares some crafting and discovery ideas with Minecraft, but the resemblance stops there. In this 2D platformer, you craft to create stronger weapons and armors so you can explore further away from your base. That’s about the extent that I know of it, I only spent an hour or so on it but I definitely want to see how far that rabbit hole goes. The developers recently announced that they were moving on, but there is already enough material in the current release to keep someone playing for a long, long time (some fans have clocked more than 100 hours on this game).

Skyrim

While I did buy Oblivion on the day it came out, I felt that I never really explored all that it had to give because I was extremely turned off by the clunky, obviously console targetted, UI. While I think that Skyrim is one of the best games I’ve played these past years, I’m sad to report that it suffers from the exact problem as Oblivion. However, the UI is not irreparably damaged that it makes playing it on a PC impossible, so after some tweaking, I was able to play the game in a reasonably satisfying manner. In one sentence, Skyrim is what World of Warcraft would be if it weren’t a multiplayer game.

I spent about thirty hours on it and reached level 20 while barely touching the main quest line (and I had a tremendous amount of fun doing so). And then my excitement and interest waned. Again, I really want to go back to it, if only to finish the main quest and visit some of the gorgeous locations I’ve heard of, such as Black Reach, so I really hope I’ll find enough resolve and time to do so.

Rift

Rift recently became free to play, so why not try it? I did but after an hour spent in the beginner zone, I can’t say I felt compelled to push further. Admittedly, this is no way to assess a MMO game, so I’m aware I might be missing out.

Kingdoms of Amalur: Reckoning

Another attempt at reusing some of World of Warcraft’s successful recipe without the MMO component. The main difference is that the fights are much more action oriented (this game also runs on consoles, after all), so you will find yourself sidestepping, dodging, rolling and also swinging, parrying and triggering various special combinations during your fights. The story line seems to be fairly expansive with a lot of content to go through. I have only spent five hours on this game and it feels like I have barely scratched its surface.

X-Com, UFO defense: Enemy Unknown

No, I didn’t get the screen shot wrong. I really mean the original X-Com game, released in 1993. I mentioned in the introduction that I would talk about another game ranked among the best of all times, and this is it.

I never really played the original when it came out but given the non stop rants and articles I have read about it during these past two decades, I have always wanted to give it a fair shot. What eventually motivated me was learning that Firaxis was going to create a sequel for it. There have been several sequels to the game but the one that the Firaxis (of Civilization fame) is working on is pretty much guaranteed to be true to the original. Here is one of the two interviews they gave on the subject. These guys mean business.

Back to the original game: it’s very simple to install and play on any OS (thanks to DOSBox) and while I expected some frustration from the punishing user interface, I ended up finding my way through it fairly easily. I only played a few hours and I managed to down my first UFO, so I can’t say that I have experienced what everyone has been raving about, but I think I now have a better idea. I’m hoping I’ll be able to push through a bit further before Firaxis’ version of the game comes out.

Ultima 6

Yes, that Ultima 6. Inspired by the riveting reports of the “CRPG Addict” blog, I decided I wanted to replay an Ultima. Since he had just reviewed Ultima 5 and I still felt that Ultima 7 was fresh in my mind (I played it a couple of years ago), I decided to go for Ultima 6. I wasn’t holding much hope, but it didn’t take long for Lord British’s creation of 1990 to completely suck me in again. Next thing I know, I was taking tons of notes and slowly unraveling the mystery of the False Prophet. It was a blast from start to finish, but unfortunately, I didn’t keep track of how many hours I spent on it. I did save all my notes for posterity (the annotated map might come in handy and it doesn’t contain any major spoilers).

Maybe I’ll carry on with Ultima 7, which has several flavors available (I’ve only ever played the first one, so I have at least three to discover before moving on to Ultima 8, which I don’t have fond memories of).

Ankh

Don’t try, you have probably never heard of this game. It was released in 1984 on the Apple ][ and even back then, it was a pretty obscure game. Yet, I remember vividly spending hours and hours on it during high school until I finally managed to beat it. For some reason, I thought about this game again recently and I decided to see how well it lived up to my memory.

Like most of the games that came out at the time, it’s pretty hard to describe and bizarre in its own way, but that’s also what makes it very interesting. You are controlling a ship in across a maze made of sixty-four rooms and your goal is to explore all of them. Part of the fun of the game is to uncover its mechanisms and how to unlock doors. Once you have solved the game, you can complete it in less than twenty minutes but to get there, you should expect to spend at most a couple of hours to unravel its mysteries. I have to say I enjoyed playing it this time around and I certainly wish we saw more of these creative and innovative games today.

The Longest Journey

I actually wrote an extensive review of this absolutely wonderful adventure game last year, so I’ll just refer you there if you are curious. Even if you’re not that much into adventure games, I strongly recommend giving this game a shot, it’s perfect in pretty much all respects and it sits up there with the likes of “Monkey Island”, “Indiana Jones and the Fate of Atlantis” and “Zak Mc Kracken and the Alien Mindbenders”.

Monkey Island 2: Le Chuck’s Revenge

Speaking of classics, I spent some time rediscovering the Monkey Island series and I play them whenever there is a bit of a lull in my video games activities. The Monkey Island Special Editions are fantastic, they talk about pirates, they are funny and the entire series is considered the best adventure game saga of all times. What’s not to like?

Bastion

An indie game that took everyone by surprise last year. The soundtrack is beautiful and it’s hard not to start paying attention to the story as it slowly unfolds through the wonderful Chandler-like narration. Bastion also sits in a niche of its own in terms of game play, with the land assembling ahead of you as you make progress through the levels. It offers various weapons (melee, range and special ability) which can be boosted by elixirs (buffs) and other customizations that you can gain either through the game or by performing well on challenge levels (such as knocking out all the enemies with only fifteen shots of your scrap musket). I must be about one third through the game and while I’m starting to feel a little bit of repetition, I really want to reach the conclusion of this game to find out where it’s taking me.

World of Warcraft: Cataclysm

“You can check out any time you want, but you can never leave”. This probably sums up my relationship with World of Warcraft since 2004. At least, I feel I made a major advance two years ago when I stopped raiding, and at that point, I thought I was pretty much done. Well, it turns out that the devils living in Blizzard’s halls know a thing or two on the multiple ways in which they can hook you to a game, and they certainly managed to pull that trick on me since I became a casual player.

The good news is that I’m no longer spending as many hours as I used to in Azeroth and also that my interest goes in bursts. The latest example is deciding to level a new character and take it to the very end of the game: killing Deathwing. I was curious to find out how long it would take, and while I thought most of it would be a grind, I actually enjoyed goign through the entire journey, probably since it was a good opportunity to find out exactly what changes had been made to the early areas of Cataclysm, but also because the new “Looking for Group” mechanism makes leveling very fast. You basically queue all the time and you then have the choice to either quest or accept the dungeon invitation.

As soon as I met the level 85 gear requirements to queue for the new “Looking for Raid” interface, I did so and one hour later, I had killed Deathwing. It took me about five days played (emphasis on “played”: that’s more than one hundred hours of game time and two-three months of real time) to take my character from level one to end game. And once I was done with this task, I… well, felt absolutely no urge to launch the game again. I’m probably done until the next expansion, the Mists of Pandaria, comes out, at which point I will probably fall for it again.

Conclusion

So this is my video game history of this past year. If there is one thing I’m taking away, it’s that I’m finding it increasingly hard to be interested in a game for extended periods of time, World of Warcraft being the striking and sole exception. I seem to hit a ceiling at twenty or thirty hours, and most of the games I play don’t even make it that far. This year, I’m looking forward to playing Mass Effect 3, Diablo 3 and maybe give Guild Wars 2 a try. Hopefully, I’ll come across a few other interesting games that are not sequels of well established franchises, something that seems all but impossible outside the Windows gaming arena.

Solution to the “Power story” challenge

Cedric — Fri, 02 Mar 2012 18:15:29 +0000

Here is the solution to the “Power story” non coding challenge:

When the power went out, Ann open the wireless settings of her laptop, scanned the area but couldn’t find a single wireless access point in the neighborhood.

This is actually based on a true story (mine, yesterday). I can usually see a dozen access points at any time, but they were all gone after the power went out yesterday, which gave me the idea for this challenge.

We had a few good answers, some that came close and a few more that answered without reading the problem carefully .

I have approved all the moderated comments on the original challenge.

Non coding challenge: a power story

Cedric — Fri, 02 Mar 2012 05:54:07 +0000

Since the next coding challenge will be a bit brutal, I thought we all earned a little break so here is a short non coding challenge. I’m pretty sure you won’t find the answer on the net since I just made it up…

Ann was enjoying this Sunday morning in her San Francisco apartment by sipping a nice cup of coffee and browsing the web on her laptop and her home wifi connection when the entire apartment went dark. Her husband entered the room:

“I might have tripped a fuse with my computer, I’ll go check the breaker box.”

“Don’t bother, it’s a general outage, the neighbors lost power too.”

“Oh? How do you know? Did you talk to them?”

“No.”

“Did you make a phone call or receive one?”

“No.”

“Did you talk to anyone?”

“Only you.”

“Did you see something outside that tipped you off?”

“No.”

How did Ann know that the outage was general and not just their apartment?

Update 1: I will hold the comments in the moderation queue for a little while to avoid spoilers, but I’ll publish them all shortly.

Update 2: I posted the solution.

Various ways to get randomness wrong

Cedric — Mon, 20 Feb 2012 17:00:59 +0000

xkcd’s random generator

Here is the wrap up of the coding challenge I proposed a few days ago, and which gathered some interesting feedback, both in the comments and also on reddit.

Randomness

First of all, as many commenters pointed out, you can’t really test for randomness, so we’ll have to settle for a looser criterion, such as even distribution. Randomness usually implies even distribution but not the other way around. On the face of it, even distribution can seem like a very weak assertion (for example, the series “[1,2,3],[2,3,1],[3,1,2],[1,2,3],…” is evenly distributed but obviously not random) but if you can ascertain it with a data set that was generated from a fair random source, then you can probably make a good claim that your algorithm is reasonably unbiased.

Shuffling

Let’s get the first question out of the way. The Fisher-Yates shuffle is very simple to explain: start from the end and swap the element at that index with a random one. Then take the next to last and swap it with a random element between 0 and i -1. Here is the (simplified) Java library version, Collections.shuffle():

public static void shuffle(List list, Random rnd) {
    int size = list.size();
    for (int i=size; i>1; i--)
        swap(list, i-1, rnd.nextInt(i));
    }
}

Note: this is actually *not* the Fisher-Yates algorithm, which runs in O(n^2) but an improved version, which runs in linear time. For some reason, most people still call it the Fisher-Yates algorithm, probably because of its resemblance to it.

Before I can show in how many ways you can actually get this wrong, we need to find out how can measure the distribution of these results, which was the heart of the second question.

Simple distribution measurements

Again, there are many ways to go about this and all of them have various compromises in time and space (mostly in space, actually). One of the most intuitive ways of reasoning about this is to maintain a matrix where each element [i,j] records how many times the digit i appeared at index j. If we call N the size of the array (for example 5) and RUNS the number of times we will invoke our function to get a representative sample (for example one million), we will generate one million arrays, so five million numbers. Since the matrix is NxN = 25, we would expect each element of the matrix to be approximately 5,000,000 / 25 = 200,000.

I will be using this approach in the rest of this article, but for the sake of completeness, let’s spend a short time considering other options.

You might be thinking that such a matrix doesn’t contain enough information. For example, while it will tell you that “2″ appeared 201,000 times in position “3″, it doesn’t tell you if there was any pattern to these appearances or whether these appearances were coupled with other numbers in the array. Whatever additional data you want to measure, you will probably end up having to extend your matrix with either constant space or maybe N (e.g. turn it into a 3D matrix).

Another thought might be that instead of counting the occurrences of these numbers, we could calculate the average of the number that each index has contained. And if you have some education in statistics, you know that once you start getting into the idea of computing averages, you can’t avoid looking at standard deviations as well.

Here is what we get for an evenly distributed algorithm:

Averages:[1.9988, 1.9992, 2.0023, 2.0011, 1.9983]
Stdev:   [1.4131, 1.4145, 1.4145, 1.4147, 1.4140]

The average is 2 since the elements range from 0 to 4 inclusive. The standard deviation is the square root of 2, which means that the numbers are evenly distributed around the average within a distance of 2.

Here are the results for the identity function, which doesn’t distribute at all:

Averages:[0.0, 1.0, 2.0, 3.0, 4.0]
Stdev:   [2.0, 1.0, 0.0, 1.0, 2.0]

Unsurprisingly, the average of numbers appearing in position 0 is 0 and 4 for the position 4. The elements in position 0 and 4 are 2 away from the average, etc…

So we can get some approximate information about how evenly distributed our function is with a simple average and standard deviation, but if we want to dig further, we need to go back to our count matrix.

The count matrix

Here is how the identity function spreads on the count matrix:

 1000000      0      0      0      0
      0 1000000      0      0      0
      0      0 1000000      0      0
      0      0      0 1000000      0
      0      0      0      0 1000000

And here is a typical run for Collections.shuffle():

2000455 2000514 2000895 1998349 1999787
2001463 1999130 1999621 1999253 2000533
1997862 2000217 2001450 2000810 1999661
1999117 1999870 1998689 2000809 2001515
2001103 2000269 1999345 2000779 1998504

This looks good but it’s a bit difficult to read, so instead, let’s count how many of these numbers differ from the expected value by, say, 1%:

for (int i = 0; i < N; i++) {
  for (int j = 0; j < N; j++) {
    float delta = Math.abs((RUNS / N) - mat[i][j]);
    float percent = delta * 100f / N / RUNS;
    if (percent > 0.01f) {
      failures++;
    }
  }
}

The outcome:

Failures: 0/25

We would expect the number of failures to increase as we reduce the number of runs:

Runs: 10000000, failures: 0/25
Runs: 1000000, failures: 0/25
Runs: 100000, failures: 15/25
Runs: 10000, failures: 24/25
Runs: 1000, failures: 24/25

Doing it wrong

Alright, so Collections.shuffle() seems to score well on this scale. Now let’s turn our attention to the many ways you can get this algorithm wrong. Let’s make a copy of it:

for (int i = N; i > 1; i--) {
  Collections.swap(result, i-1, rnd.nextInt(N));
}

Let’s reset RUNS to one million and try again:

Runs: 1000000, failures: 17/25

Oops. Running it again and again yields similar results. What happened?

Okay, I’ll admit it, I introduced a bug when copying the code, can you spot it?

  Collections.swap(result, i-1, rnd.nextInt(N));  // incorrect
  Collections.swap(result, i-1, rnd.nextInt(i));  // correct

It looks like selecting the random index to swap among the entire array totally wrecks the distribution, while selecting it from only these elements we haven’t reached in our loop makes everything right. I’ll leave the interpretation of this behavior as an exercise to the reader.

Here is another way to get this wrong.

When you look at the fundamentals of the algorithm, you may wonder if you really need to go through the entire array to perform the swaps. After all, once you reach the half point of the array, it’s very likely that the second half of the array has been swapped randomly already, so why not stop there? Well, let’s try (I am also increasing the size of the array to 10):

Runs: 1000000, failures: 25/100

Again, it looks like we messed up the algorithm. Let’s take a look at the distribution matrix:

    for (int i = N; i > N / 2; i--) {
      Collections.swap(result, i-1, rnd.nextInt(i));
    }

 499916      0      0      0      0  99893 100776  99757  99744  99914
      0 499489      0      0      0 100013 100040 100580  99643 100235
      0      0 499929      0      0  99719  99753 100135 100363 100101
      0      0      0 500268      0  99903 100346  99934 100079  99470
      0      0      0      0 500255  99669  99766 100426 100040  99844
 100364 100323  99846  99878  99909 100102  99518  99608  99934 100518
 100392  99868  99843 100204  99662 100333  99544  99582 100313 100259
  99946 100439 100138  99926 100406  99785 100144  99898  99773  99545
  99906  99622  99748 100211  99641 100624 100149 100188 100003  99908
  99476 100259 100496  99513 100127  99959  99964  99892 100108 100206

Our modified algorithm has introduced a strong bias in the first half of the array (the one we decided wasn’t worth selecting on): number 0 has a 50% chance of being found in position 0, same for 1-4. From 5 to 9, the probability drops to the expected 10% odds.

Let’s see what happens if we go a bit further than the half point, say 3/4 of the array:

    for (int i = N; i > N / 4; i--) {
      Collections.swap(result, i-1, rnd.nextInt(i));
    }

 199836      0 100150 100374 100292 100139  99861  99665  99861  99822
      0 200711 100183  99753 100317  99945  99761  99403  99879 100048
 100283  99860  99936  99915  99693 100480  99786 100128  99848 100071
 100057  99592 100152  99785 100017  99507  99550 100557 100564 100219
  99933 100225  99808 100193  99844 100017 100212  99573 100462  99733
  99776  99789  99895  99753  99953  99973 100124 100873  99518 100346
  99921 100068 100020 100316  99589  99728 100461 100206  99900  99791
 100169  99896  99941  99880  99922 100094 100168  99947  99754 100229
 100217  99964  99812 100066 100413 100209  99975  99593 100216  99535
  99808  99895 100103  99965  99960  99908 100102 100055  99998 100206
Runs: 1000000, failures: 4/100

Here is another way to mess it up. Let’s restore our loop to swap the entire array:

    for (int i = N; i > 1; i--) {
      Collections.swap(result, i-1, new Random().nextInt(i));
    }

Let’s run it:

 137244  44545  91552  87602 100969 122467 100299 125866 101490  87966
  71720 138821  92466 107337 103311 123042  99610  73350 103791  86552
 129376 113794 114303  98828 102760  55589 100312 102734  95621  86683
  71216 112960 111268 115538 102947  99968 100271  98012 100026  87794
  94519  94294  94709  94851  93888 103046  99956 100257 101460 123020
  95533  96387  95852  96858  96351  96101  99570  99488  96360 127500
  99901  99926  99890  99421  99929 100409  99989  99744 100489 100302
 100233  99852  99969  99940  99872  99922 100016 100154 100071  99971
  99725  99387  99817  99841 100002  99500 100061 100811 100863  99993
 100533 100034 100174  99784  99971  99956  99916  99584  99829 100219
Runs: 1000000, failures: 49/100

Wow, that’s another massive failure. What did I do wrong this time?

    private Random rnd = new Random();

// ...

    Collections.swap(result, i-1, rnd.nextInt(i));  // correct
    Collections.swap(result, i-1, new Random().nextInt(i)); // incorrect

Why would creating a new Random object trigger such a bug while reusing the same Random object through our loops provides a good distribution?

If you ever deal with random generation in Java (and probably other languages), you absolutely need to be aware of the way java.lang.Random is implemented. The problem here is that new objects are seeded with the current time, so creating objects within a loop pretty much guarantees that these objects will be created within a timeframe that’s smaller than the clock granularity, which means that a lof ot the random numbers generated in succession will be identical. You address this either by reusing the same Random object (consider warming it up, too) or, better, by using java.util.Math#random (or even better: java.security.SecureRandom).

I’ll show one more common mistake, which might actually not always be one:

    for (int i = N; i > 1; i--) {
      Collections.swap(result, i-1, rnd.nextInt(i - 1));  // was i
    }

The difference here is that the random index is picked between 0 and i-1, instead of between 0 and i. Here is the distribution:

      0 249684 250515 249406 250395
 250334      0 249828 249915 249923
 249569 249612      0 250188 250631
 249973 250505 250471      0 249051
 250124 250199 249186 250491      0
Runs: 1000000, failures: 25/25

We observe that no number i can ever be found in position i, which, while totally failing our even distribution test, possesses a few interesting qualities. This approach is actually known as the Sattolo algorithm, and it’s actually useful because it always generates cycles (look it up in a section of its own in the Fisher-Yates entry on Wikipedia).

This is getting much longer than I anticipated, so I’ll just leave you with a final thought in the form of a simple problem, which touches on a question that Lawrence Kesteloot mentioned on in his comment on the original entry: I’m giving you an unbiased random source that returns either 0 or 1 and I want you to use it to generate a random list of the letters ‘a’, ‘b’, ‘c’, ‘d’ and ‘e’. And then, extend the idea to shuffle a deck of 52 cards.

Update: discussion on reddit

A new coding challenge

Cedric — Thu, 16 Feb 2012 20:43:24 +0000

It’s been a while since I proposed a coding challenge, so I’ve been thinking about another one. Coming up with original problems that are not solved with a simple Google search is not easy, but I think this new one, while not especially hard, should produce some interesting and creative results.

This problem comes in two parts. The first part is more of a warm up since it’s very easy to look up, but the follow up question will, hopefully, be the source of innovative answers.

Here it is:

1) Write a function which, passed an int n, returns an array of size n containing all the numbers between 0 and n-1 in random order. For example, with n=5, valid answers are [0, 2, 3, 1, 4], [4, 1, 2, 0, 3], etc…

2) Prove that the function you wrote in 1) returns “really random” arrays.

I’m being intentionally vague on how to answer the second question in order not to lead the answers, but hopefully, the question is specific enough that no further clarifications is needed. Feel free to ask in the comments otherwise.

All languages welcome, and I suggest you use pastebin to submit your code.

The only sure way to lose is not to play

Cedric — Thu, 09 Feb 2012 18:24:31 +0000

Google’s recent letter to the IEEE about their position on the current lawsuits between Motorola Mobility and Apple seems to spark some outrage across the mobile land. And I’m puzzled.

First of all, I don’t think any specific company is innocent in this dance, starting with Apple themselves. In their letter to the ETSI, Apple stated (quoted):

Apple’s letter then moves on to propose a solution based on three specific principles: appropriate royalty rate; common royalty base; no injunction.

That’s a nice thought, but it’s making quick work of Apple’s recent attempts at obtaining various injunctions against its rivals in the recent past, most notably Samsung.

So what’s going on, here, now that the injunction course of action didn’t work for them, Apple thinks injunctions should be disallowed for everyone?

This whole patent war is ugly. Ugly, ugly, ugly. When Apple fired the first shot against Samsung in April 2011, the entire community seemed to be in wide agreement that this was a crazy move, one that could only lead to the mutually assured destructions of all the parties involved. Obviously, Apple had weighed these risks and they came to the conclusion that either they could afford the collateral or that this collateral would be very small anyway, since they thought that innovation was on their side.

As the events since April have shown, things are obviously not so clear cut and innovation seems to be spread quite evenly among all the parties involved. The result of Apple’s opening salvo has been a flurry of suits and counter suits which, frankly, have led nowhere. All the companies involved are poorer than before, their patent attorneys are much richer and the whole field is an intellectual property war zone.

In the letter mentioned above, Google is simply stating that they won’t change anything to the legal actions that Motorola Mobility had underway before Google acquired them. That’s it.

It makes as much sense getting upset about this as it is to be outraged at someone because they play chess viciously.

Now that the war is full on, I can’t find anything wrong with all the players trying to fight as much as they can and let the courts decide who’s right and who’s wrong. The only stupid move in this conflict would be not to play.

Ushering TestNG into the new year

Cedric — Mon, 02 Jan 2012 04:35:48 +0000

I took advantage of the holiday break to completely revamp TestNG’s HTML reports, something I’ve been meaning to do for a very long time but never found the time to put at the top of my TODO list. I wrote the original reports pretty much with the very first version of TestNG, around 2004, and I hardly touched them since then. My intention with this rewrite was not just to revamp them visually but also technically, so that I could give myself as much freedom as possible to improve them from this point on.

Here are some notes I took during this process.

Content and appearance

First of all, here are the new reports. They show all the suites on the same page with a banner at the top, a navigator pane on the left hand side (which always stays visible) and a main panel on the right hand side, which shows all the information you requested depending on which item you clicked in the navigator. Pretty straightforward.

If you take a look at the source, you will see that it hardly contains any HTML: it’s mostly made of

and . I didn’t really set out to do this initially, it’s just that whenever I used HTML elements, I inherited undesirable CSS attributes (margins, paddings, etc…) which I ended up resetting manually, so after a while, it just seemed much easier to use divs and spans knowing that they start with a clean CSS slate.

Since there is so much generated text, I pondered using a templating library to make my life easier. The first question to settle was: client side or server side templates? In this case, there is not really a “server” side, so the question is more “Java” or “Javascript” templates. I quickly rejected the client side approach which, while well adapted to serve pages over a network, doesn’t seem to have much benefit in this particular case since the page is generated locally. As for the Java side, the two standard options for templates (Freemarker and Velocity) seemed quite overkill for what I was trying to do, so I ended up writing my own tiny implementation of mustache.js (which I’ve used in the past and liked quite a bit)..

Once I had the templating code working, I started converting the Java code over to it but… something felt wrong. I quickly gained the impression that this was actually a step backward compared to doing the generation 100% from Java, and the reason quickly became apparent to me: the Java compiler.

The problem with using templates is that you get very little help from the tools. Some template libraries allow you to include other templates, which can reduce the repetition, but it’s still very easy to make simple typos or being forced to copy/paste a lot. In contrast, my Java hierarchies makes it quite trivial to, say, add a new panel on the right hand side with a link to the left. Implement the right class, declare it and it will automatically work, with the right defaults (CSS selectors) and a lot of the logic validated by the compiler.

This comfort was too good to pass up so I scrapped my template idea (my mustache.js implementation is still around, though, and I might use it some time in the future) and I stuck to 100% Java generation.

Javascript and development tools

Picking jQuery was a no-brainer, it’s hard for me to imagine doing anything in Javascript that manipulates the DOM without using JQuery. The logic in the new reports is fairly simple and is just above one hundred lines of Javascript.

Chrome’s development environment is also quite superb (I’m sure other browsers’ is just as good). In a nutshell, you have access to pretty much the same range of support that Eclipse or IDEA provides in Java: breakpoints, inspection and modification of variables, and even CSS debugging, which I’ve found invaluable to track down unexpected CSS behaviors (I came across quite a few).

If you haven’t kept up with what you can do with Chrome/Javascript/JQuery these days, try the following:

Open http://testng.org/new in Chrome.
Right click anywhere on the page and select "Inspect element". This will open the document explorer at the bottom.
With the focus in the document explorer, type ESC, which will open the REPL.
Type $('.navigator-suite-content').hide().
This will collapse the content under both suites.

You can see how easy it is to debug this kind of code.

Other libraries

If you select the "Times" link of the first test suite in the Navigator pane, you will see the following table:

Click to enlarge

I am using Google’s Chart Tools to display this table and it was quite straightforward to add. The only tricky part was how to generate the data in Javascript form and make sure that the table gets drawn only after this data has been initialized, which required using a few Javascript tricks to make sure the initialization order is correct. The Google Chart Tools are very powerful and I will probably use more of their API to display additional graphs in the TestNG reports (pie charts, etc…).

Conclusions

Here are my main take away points:

CSS still feels like black magic to me. The theory is trivial on paper but in reality, I often come across results that just don’t make sense. With the help of modern CSS debuggers, it’s a little bit easier to find out what is happening and why (I especially like Chrome’s “Computed style” which gives you a list of all the attributes that were derived from .css files), but there are still times where I feel completely helpless trying to find out why something is not how it should be.
I’m a bit concerned with the size of the reports: I was quite surprised to find out that the reports you’re looking at is one meg in size. Admittedly, it shows over five hundred test methods, but the fact that it contains all the panels, even those the user might not be interested in, is a potential place for improvement. I might decide to put each individual pane in its own file.
Javascript’s rubbery type system is handy for this kind of task, but I’m still a bit unsure how well it scales to hundreds of thousands of lines of code. For example, I really enjoy being able to say if (v) regardless of what type v is and have Javascript’s truthiness values do the right thing. Similarly, it’s nice to be able to add a parameter to a function and not having to update one single call site (if you call a Javascript method with less parameters than it expects, the extra parameters simply receive the Undefined value. Obviously you need to test for this in said function).
JQuery is great and it’s probably making Javascript even more popular than it already is. I’m hoping the Dart team is taking notes and making sure that a similarly powerful and elegant framework will be available in Dart.

Finally, a call for help: I welcome any feedback on how to improve the CSS of these new reports, so if you are so inclined, feel free to improve the look of these new reports and share your improvements with me. I will be very grateful.

Happy new year!

Ground control to major Tom

Cedric — Tue, 13 Dec 2011 07:55:11 +0000

This could be you!

Who hasn’t dreamed of becoming an astronaut? Well, now you can. The NASA has an open position for an “Astronaut Candidate” for the International Space Station in Houston, TX. The requirements are pretty interesting.

The training will take two to three years and will require a lot of traveling, probably mostly to Russia, since they will be using Soyuz to travel to and from the ISS. The required education is fairly broad, and more interestingly, the list of degrees that are not applicable is fairly short (technology, psychology, nursing(!), etc…). It looks like a CS degree is good enough, and you can make up for the lack of flying experience with an MS or better, a PhD.

“Correctable” 20/20 eyesight is required, which means that contact lenses and Lasik are accepted (I wonder how hard it is to put contact lenses on in zero gravity…). The ad also takes the time to explain each jargon occurrence: “Extravehicular Activities (space walks)”, “the extravehicular activity mobility unit (space suit)”.

Although the ad will only be open for the next couple of months, the selection process takes a while since the selection will be announced in the spring of 2013.

I was fortunate to be able to experience a Zero-G flight a few years ago, I can’t say I’m not tempted to apply