The Kobalt diaries: Android

A lot of work has gone into Kobalt since I announced its alpha:

I’m plannning to post more detailed updates as things progress but today, I’d like to briefly show a major milestone: the first Android APK generated by Kobalt.

I picked the Code path intro app as a test application. I first built it with Gradle to get a feel for it and the apk was generated in about 27 seconds. Then I generated a Build.kt file with ./kobaltw --init, added a few Android related directives to reach the following (complete) build file:

import com.beust.kobalt.*
import com.beust.kobalt.plugin.android.*
import com.beust.kobalt.plugin.java.*

val p = javaProject {
    name = "intro_android_demo"
    group = "com.example"
    artifactId = name
    version = "0.1"

    dependencies {
        compile("com.android.support:support-v4:aar:23.1",
                file("app/libs/android-async-http-1.4.3".jar))
    }

    sourceDirectories {
        listOf(path("app/src/main/java"))
    }

    android {
        applicationId = name
        buildToolsVersion = "21.1.2"
    }
}

Then I launched the build with ./kobaltw assemble, and…

Less than five seconds to generate R.java, compile it, compile the code, run aapt, generate classes.dex and finally, generated the apk. If you are curious, you can check out the full log.

Admittedly, Kobalt doesn’t yet handle build types and flavors nor manifest merging, but the example app I’m building here doesn’t use those either so I don’t expect the build time to increase much. There is a lot more to be done before Kobalt’s Android plug-in is ready for more users, but this is a pretty encouraging result.

Exploring the Kotlin standard library (part 2)

I folded the two parts of this series into one blog post, which you can read here.

Exploring the Kotlin standard library

Standard.kt is part of the Kotlin library and it defines some essential functions. What’s really striking about this source file is that it’s less than fifty lines long and that each of the function it defines (less than ten) is a one liner. Yet, each of these functions is very powerful. Here is a quick overview of the most important ones.

let()

fun <T, R> T.let(f: (T) -> R): R = f(this)

let() is a scoping function: use it whenever you want to define a variable for a specific scope of your code but not beyond. It’s extremely useful to keep your code nicely self-contained so that you don’t have variables “leaking out”: being accessible past the point where they should be.

DbConnection.getConnection().let { connection ->
}
// connection is no longer visible here

let() can also be used as an alternative to testing against null:

val map : Map<String, Config> = ...
val config = map[key]
// config is a "Config?"
config?.let {
    // This whole block will not be executed if "config" is null.
    // Additionally, "it" has now been cast to a "Config" (no question mark)
}

apply()

fun <T> T.apply(f: T.() -> Unit): T { f(); return this }

apply() defines an extension function on all types. When you invoke it, it calls the closure passed in parameter and then returns the receiver object that closure ran on. Sounds complicated? It’s actually very simple and extremely useful. Here is an example:

File(dir).apply { mkdirs() }

This snippet turns a String into a File object, calls mkdirs() on it and then returns the file. The equivalent Java code is a bit verbose:

File makeDir(String path) {
  File result = new File(path);
  result.mkdirs();
  return result;
}

apply() turns this kind of ubiquitous code into a one liner.

with()

fun <T, R> with(receiver: T, f: T.() -> R): R = receiver.f()

with() is convenient when you find yourself having to call multiple different methods on the same object. Instead of repeating the variable containing this object on each line, you can instead “factor it out” with a with call:

val w = Window()
with(w) {
  setWidth(100)
  setHeight(200)
  setBackground(RED)
}

run()

fun <T, R> T.run(f: T.() -> R): R = f()

run() is another interesting one liner from the standard library. Its definition is so simple that it looks almost useless but it’s actually a combination of with() and let(), which reinforces what I was saying earlier about the fact that because all these functions from the standard library are regular functions, they can be easily combined to create more powerful expressions.

Tying it all together

Of course, it’s actually possible (and encouraged) to combine these functions:

fun configurationFor(id: String) = map[id]?.let { config ->
  config.apply {
    buildType = "DEBUG"
    version = "1.2"
  }
}

This code looks up a Config object from an id and if one is found, sets a few additional properties on it and then returns it. But we can simplify this code even further. This time, I’m providing a fully self-contained snippet so you can copy and paste it directly into Try Kotlin in order to run it yourself:

class Config(var buildType: String, var version: String)

val map = hashMapOf<String, Config>()

fun configurationFor(id: String) = map[id]?.let { config ->
    config.apply {
        buildType = "DEBUG"
        version = "1.2"
    }
}

Don’t you feel that this combination of let() and apply() feels a bit boilerplatey? Let’s rewrite it a bit more idiomatically:

fun configurationFor(id: String) = map[id]?.apply {
    buildType = "DEBUG"
    version = "1.2"
}

Let’s unpack this rather dense snippet:

  • Looking up a value on a hash map can be done either with get() or with the bracket notation, which is preferred.
  • Since the key might not be present in the map, we use the safe dereference operator ?. which guarantees that we will only enter apply() if the result is non null.
  • Inside the apply() block, the this object is a Config, which lets us invoke functions on this object without any prefix. In this case, all we have is properties, but obviously, you could invoke regular functions just as well.
  • Once that code has run, the altered Config is returned.

use()

fun <T : Closeable, R> T.use(block: (T) -> R): R

Another interesting function of the standard library is use(), which gives us the equivalent of Java’s try-with-resources and of C#’s using statement.

This function applies to all objects of type Closeable and it automatically closes its receiver on exit. Note that as opposed to Java and C#, Kotlin’s use() is a regular library function and not directly baked in the language with a special syntax. This is made possible by Kotlin’s extension functions and closure syntax used coinjointly.

// Java 1.7 and above
Properties prop = new Properties();
try (FileInputStream fis = new FileInputStream("config.properties")) {
    prop.load(fis);
}
// fis automatically closed
// Kotlin
val prop = Properties()
FileInputStream("config.properties").use {
    prop.load(it)
}
// FileInputStream automatically closed

Because Kotlin’s version is just a regular function, it’s actually much more composable than Java’s. For example, did you want to return this prop object after loading it?

// Kotlin
fun readProperties() = Properties().apply {
    FileInputStream("config.properties").use { fis ->
        load(fis)
    }
}

The apply() call tells us that the type of this expression is that of the object apply() is invoked on, which is Properties. Inside this block, this is now of type Properties, which allows us to call load() on it directly. In between, we create a FileInputStream that we use to populate this property object. And once we call use() on it, that FileInputStream will be automatically closed before this function returns, saving us from the ugly try/catch/finally combo that Java requires.

You will find a lot of these constructs in the Kobalt build tool code, feel free to browse it.

Google Fi unboxing



I received my order of Google Fi, the package contained more than I expected.



The business card size item at the bottom is the SD card. The rest is:

  • A portable charger.
  • A case for your Nexus 6.
  • A headset.



The charger has two USB ports and a micro one. Apparently, you can charge it from any of these ports (very convenient) and then you can plug two phones at the same time (probably three if you can find a dual micro-USB cable).



Finally, the headset has something that’s hard to find in headsets in general: volume control. It also has an extra jack, so you can plug another heaset in it. The only downside of this headset is that the control block dangles on your cheek instead of being located much lower on the cable. I don’t understand why such headsets are still manufactured.

I haven’t tested the service yet, I’ll report back after I’ve had a chance to use it thoroughly.

The long and arduous road to JCenter and Maven bliss

TestNG is available on both Maven Central and JCenter and I used to publish the artifact in these two repos with Maven. Recently, I took some time trying to obtain the same result with Gradle and so far, it has been a very painful and agonizing experience because there are so many moving parts to the whole process:

  • Gradle itself.
  • Using the right plugins, then configuring them properly.
  • Understanding the intricacies of JCenter/Bintray publishing.
  • Too much incorrect information out there. There are a lot of tutorials available on the Internet, way too many actually, especially since a lot of them are out of date.
  • IDEA is offering close to no help while editing your Gradle file: no auto completion, claiming your file has errors when it doesn’t, not complaining about broken files, etc…

My goal getting into this operation was simple and, so I thought, reasonable: being able to publish snapshots and releases from the command line to both JCenter and Maven. So far, my conclusion is that there is really no simpleem> way to achieve this goal. There is a complicated way, which I describe below, but even that complicated way doesn’t quite achieve my goal. In the end, I’m getting close to that goal except that I’m still going manually through the Nexus UI to close and deploy the artifact to Maven Central. I’ll post an update if I ever solve this.

The final structure of the build layout looks like this:

  • build.gradle: main build file, which at the end includes
  • gradle/publishing.gradle: sets up publishing routines and values that are shared by both Maven Central and JCenter. At the end, this script includes
  • gradle/publishing-maven.gradle and gradle/publishing-jcenter.gradle, which include the respective plugins and perform the publishing.

What follows is not a tutorial (there are so many alrady) but instead, a series of errors that I encountered along the way and how I fixed them.

401 when uploading to bintray

Verify your credentials. One way of setting them is:

bintray {
    user = properties.getProperty("bintray.user")
    key = properties.getProperty("bintray.apikey")

I put these values in local.properties, which I load explicitly (gradle.properties might be a better location):

bintray.user=...
bintray.apikey=...

Javadocs are not being published

Maven Central will reject artifacts that don’t contain Javadocs and these are typically not included by default:

task javadocJar(type: Jar, dependsOn: javadoc) {
    classifier = 'javadoc'
    from 'build/docs/javadoc'
}

task sourcesJar(type: Jar) {
    from sourceSets.main.allSource
    classifier = 'sources'
}

artifacts {
    archives jar
    archives javadocJar
    archives sourcesJar
}

Javadocs are not being uploaded

One line to add to your bintray configuration:

bintray {
    // Without this, javadocs don't get uploaded
    configurations = ['archives']
}

“Cannot create task of type ‘Jar’ as it does not implement the Task interface.”

At some point during my tribulations, I started encountering this mystifying error. I ended up realizing that IntelliJ had sneakily added an import org.apache.tools.ant.taskdefs.Jar at the top of my gradle file, which is obviously not the class that we want. Removing this import fixed the problem (and you might want to configure IDEA to exclude it from your imports to avoid this problem in the future).

Artifacts are not being signed

Add the following to your signing configuration:

apply plugin: 'signing'

signing {
    required { gradle.taskGraph.hasTask("bintrayUpload") }
    sign configurations.archives
}

Then in your gradle.properties (not local.properties):

signing.keyId=...
signing.password=...
signing.secretKeyRingFile=(path to .gnupg/secring.gpg)

.asc files are not being generated

Another requirement from Maven Central, which you fix in the bintray configuration:

bintray {
    pkg {
        version {
            gpg {
                // Without this, .asc files don't get generated
                sign = true
            }

“Return code is: 400, ReasonPhrase: Bad Request”

In my initial attempts, ./gradlew publish would fail with this error, probably one of the most frustrating things about Sonatype: the HTTP errors are completely opaque and they don’t give you any detail on why they failed, while they easily could. Here is a list of potential reasons for this 400:

  • user credentials are wrong
  • url to server is wrong
  • user does not have access to the deployment repository
  • user does not have access to the specific repository target
  • artifact is already deployed with that version if it is a release (not -SNAPSHOT version)
  • the repository is not suitable for deployment of the respective artifact (e.g. release repo for snapshot version, proxy repo or group instead of a hosted repository)

Just to list a few. While the server won’t give more details is beyond me and one of the main reasons why I wish I could stop dealing with Sonatype completely.

./gradlew uploadArchives” failing with mysterious HTTP errors

Another mistake I initially made was to try to upload the artifacts directly to the Maven Central repo instead of Sonatype’s Nexus staging host. The correct configuration is:

uploadArchives {
    repositories {
        mavenDeployer {
            repository(url: "https://oss.sonatype.org/service/local/staging/deploy/maven2") {
                authentication(userName: System.getenv('SONATYPE_USER'), password: System.getenv('SONATYPE_PASSWORD'))
            }
            snapshotRepository(url: "https://oss.sonatype.org/content/repositories/snapshots") {
                authentication(userName: System.getenv('SONATYPE_USER'), password: System.getenv('SONATYPE_PASSWORD'))
            }

As I said at the top of this article, you then need to go deploy the archive manually from the Nexus UI.

I can’t find an answer to my Gradle problem!

Here is a life pro tip: whenever you do Google searches about Gradle, restrict the results to only the last year and read the StackOverlow answers first. Anything published before is pretty much guaranteed to be out of date.

Sonatype documentation is terrible.

Yes, yes it is. For example, the first hit to learn how to deploy to Maven Central from Gradle will land you here. This article is actually an indirection to the “real” article here which is… 404. The next link is also a 404.

Extremely frustrating.

Next steps

My current configuration enables the following process:

  • ./gradlew bintrayUpload uploads the release to JCenter. It will fail if the version is a SNAPSHOT (intentional since I uploads the snapshots to Maven’s snapshot repo, this part is straightforward and fully automated). If you want to publish snapshots to JCenter as well, you can do this by publishing to JFrog, although my attempts in that direction have never succeeded.
  • ./gradlew uploadArchives will upload the snapshot to Maven’s snapshots repo and the release to Sonatype’s staging repo. This is decided automatically based on whether the version name contains the string “SNAPSHOT”.

The build files themselves add up to more than 300 lines, which is mind boggling to perform operations that should be close to standard. Gradle is certainly very far from having sensible defaults.

I’m hoping to eventually be able to fully deploy to Maven Central from the command line but I’m not sure it’s possible, so suggestions are welcome.

Easily inspect your SQLite database on Android

Here is a script I use very often for Android development: this small shell script will copy the database from your device on your file system and then launch SQLiteBrowser on it, allowing you to inspect your tables very quickly. I’ve found this script extremely useful, going sometimes as far as calling it multiple times while my code in on breakpoints in my IDE.

This script takes additional steps before pulling the database, such as changing a few file permissions, since I have noticed that some devices are more strict about allowing database pulling than others. As far as I can tell, this script has worked on every single device I’ve used so far.

#
# pull-db
# Inspect the database from your device
# Cedric Beust
#

PKG=com.beust.example
DB=my.db

adb shell "run-as $PKG chmod 755 /data/data/$PKG/databases"
adb shell "run-as $PKG chmod 666 /data/data/$PKG/databases/$DB"
adb shell "rm /sdcard/$DB"
adb shell "cp /data/data/$PKG/databases/$DB /sdcard/$DB" 

rm -f /tmp/${DB}
adb pull /sdcard/${DB} /tmp/${DB}

open /Applications/sqlitebrowser.app /tmp/${DB}

Game design and game implementation

“Because of a bug (it is an off-by-one error) the parley can only work if enemy and party member *do not* speak the same language.”

One of the many memorable quotes from a fascinating article on an old school RPG by one of the developers. I don’t even know this particular video game, even though I was playing a lot of RPG during my Apple ][ and Amiga days, but this particular quote resonates with me because it ties game design and game implementation together in a very explicit way.

The whole article is a must read for anyone who’s interested in game development or game design, or both.

Easy SQLite on Android with RxJava

Whenever I consider using an ORM library on my Android projects, I always end up abandoning the idea and rolling my own layer instead for a few reasons:

  • My database models have never reached the level of complexity that ORM’s help with.
  • Every ounce of performance counts on Android and I can’t help but fear that the SQL generated will not be as optimized as it should be.

Recently, I started using a pretty simple design pattern that uses Rx to offer what I think is a fairly simple way of managing your database access with RxJava. I’m calling this the “Async Rx Read” design pattern, which is a horrible name but the best I can think of right now.

Easy reads

One of the important design principles on Android is to never perform I/O on the main thread, and this obviously applies to database access. RxJava turns out to be a great fit for this problem.

I usually create one Java class per table and these tables are then managed by my SQLiteOpenHelper. With this new approach, I decided to extend my use of the helper and make it the only point of access to anything that needs to read or write to my SQL tables.

Let’s consider a simple example: a USERS table managed by the UserTable class:

// UserTable.java
List<User> getUsers(SQLiteDatabase db, String userId) {
  // select * from users where _id = {userId}
}

The problem with this method is that if you’re not careful, you will call it on the main thread, so it’s up to the caller to make sure they are always invoking this method on a background thread (and then to post their UI update back on the main thread, if they are updating the UI). Instead of relying on managing yet another thread pool or, worse, using AsyncTask, we are going to rely on RxJava to take care of the threading model for us.

Let’s rewrite this method to return a callable instead:

// UserTable.java
Callable<List<User>> getUsers(SQLiteDatabase db, String userId) {
  return new Callable<List<User>>() {
    @Override
    public List<User> call() {
      // select * from users where _id is userId
    }
  }
}

In effect, we simply refactored our method to return a lazy result, which makes it possible for the database helper to turn this result into an Observable:

// MySqliteOpenHelper.java
Observable<List<User>> getUsers(String userId) {
  return makeObservable(mUserTable.getUsers(getReadableDatabase(), userId))
    .subscribeOn(Schedulers.computation()) // note: do not use Schedulers.io()
}

Notice that on top of turning the lazy result into an Observable, the helper forces the subscription to happen on a background thread (the computation scheduler here; do not use Schedulers.io() because it’s backed by an unbounded executor). This guarantees that callers don’t have to worry about ever blocking the main thread.

Finally, the makeObservable method is pretty straightforward (and completely generic):

// MySqliteOpenHelper.java
private static <T> Observable<T> makeObservable(final Callable<T> func) {
  return Observable.create(
      new Observable.OnSubscribe<T>() {
          @Override
          public void call(Subscriber<? super T> subscriber) {
            try {
              subscriber.onNext(func.call());
            } catch(Exception ex) {
              Log.e(TAG, "Error reading from the database", ex);
            }
          }
    });
}

At this point, all our database reads have become observables that guarantee that the queries run on a background thread. Accessing the database is now pretty standard Rx code:

// DisplayUsersFragment.java
@Inject
MySqliteOpenHelper mDbHelper;

// ...

mDbHelper.getUsers(userId)
  .observeOn(AndroidSchedulers.mainThread())
  .subscribe(new Action1<List<User>>()) {
    @Override
    public void onNext(List<User> users) {
      // Update our UI with the users
    }
  }
}

And if you don’t need to update your UI with the results, just observe on a background thread.

Since your database layer is now returning observables, it’s trivial to compose and transform these results as they come in. For example, you might decide that your ContactTable is a low layer class that should not know anything about your model (the User class) and that instead, it should only return low level objects (maybe a Cursor or ContentValues). Then you can use use Rx to map these low level values into your model classes for an even cleaner separation of layers.

Two additional remarks:

  1. Your Table Java classes should contain no public methods: only package protected methods (which are accessed exclusively by your Helper, located in the same package) and private methods. No other classes should ever access these Table classes directly.

  2. This approach is extremely compatible with dependency injection: it’s trivial to have both your database helper and your individual tables injected (additional bonus: with Dagger 2, your tables can have their own component since the database helper is the only refence needed to instantiate them).

This is a very simple design pattern that has scaled remarkably well for our projects while fully enabling the power of RxJava. I also started extending this layer to provide a flexible update notification mechanism for list view adapters (not unlike what SQLBrite offers), but this will be for a future post.

This is still a work in progress, so feedback welcome!

Anko for Android

I’m really excited to see JetBrains continue to invest in Android. Not long ago, they release Anko, a library that uses Kotlin’s DSL functionalities to offer a different way to define your layouts than XML:

verticalLayout {
    val name = editText()
    button("Say Hello") {
        onClick { toast("Hello, ${name.text}!") }
    }
}

To be honest, I’m still a bit skeptical of the value that such a DSL provides for layout definitions (the XML file based layout that Android supports is extremely flexible to manage layouts for different displays) but JetBrains just released a new version of Anko and the direction this library is taking is now becoming much more interesting to me.

With the latest additions, Anko is widening its scope beyond layout definition and adding a set of helper functions that make Android development on Kotlin even cleaner and more concise. The combination of extension functions and the DSL notation really shine for Android development. Here are a few examples:

val flowers = listOf("Chrysanthemum", "Rose", "Hyacinth")
selector("What is your favorite flower?", flowers) { i ->
    toast("So your favorite flower is ${flowers[i]}, right?")
}

Doing away with the pesky TAG requirement for logging:

class SomeActivity : Activity(), AnkoLogger {
    fun someMethod() {
        info("Info message")
        debug(42) // .toString() method will be called automatically
    }
}

Check out the announcement for more snippets, or head directly to Anko’s project page.

The case of the buggy executor

I spent a mystifying half hour chasing down a bug recently, so I thought I would share.

Here is a simple scheduled executor:

public class Exec {
    final static Logger logger = LoggerFactory.getLogger(Exec.class);

    private static final ScheduledExecutorService executor =
        Executors.newSingleThreadScheduledExecutor(
            new ThreadFactory() {
                @Override
                public Thread newThread(Runnable r) {
                    Thread result = new Thread();
                    result.setName("BuggyExecutor");
                    return result;
                }
            });

    public static void main(String[] args) {
        executor.scheduleAtFixedRate(() -> logger.info("Tick"),
                0,
                1, TimeUnit.SECONDS);
    }
}

I always use a ThreadFactory in my executors since seeing thread names plainly in your trace simplifies debugging threading issues considerably. Whenever I see a pool-2-thread-3 in my thread dump, I track down the lazy library that caused that monstrosity and I seriously consider replacing it with one that is written by developers more respectful of my time.

Other than that, this code is pretty straightforward and if you run it, you would expect the string “Tick” to be displayed every second:

17:37:18.780 [BuggyExecutor] INFO  com.beust.Exec - Tick
17:37:19.780 [BuggyExecutor] INFO  com.beust.Exec - Tick
17:37:20.780 [BuggyExecutor] INFO  com.beust.Exec - Tick

However, if you run the code as provided above, you will see that it does absolutely nothing.