Painless, type-safe config file parsing with Macwire and Ficus

Mikołaj Koziarkiewicz

Intro

Scala has a distinct advantage of an "official" configuration file format, i.e. HOCON. Even better, the niceness transcends the officialdom - HOCON is quite well-defined, and it has lots of useful of features. In other words, it’s simply a good format.

Of course, any Scala developer past their Day 3 can tell you of a certain little quirk - the reference parser implementation is a Java library. Even though Java-Scala interop is actually mostly tolerable once one learns the ropes, it still remains slightly annoying.

This post will describe a possible approach for reducing that annoyance by synergizing two well-known libs.

The following is intended for devs already minimally familiar with Macwire. If you’re not, take a look at the README. Reading the the Introduction will suffice.

Config

There are several popular pure-Scala HOCON parsing libraries available. We’re going to use Ficus.

As a library, it’s extremely straightforward - specify the config file, the subpath within it, the type you want it to be parsed into - the lib will do the rest. It even supports extraction into semi-arbitrary data types, like so:

server {
    host: "example.com"
    port: 1024
  }
case class ServerConfig(host: String, port: Int)

import net.ceedubs.ficus.Ficus._
import net.ceedubs.ficus.readers.ArbitraryTypeReader._

ConfigFactory.load().as[ServerConfig]("server")
//emits ServerConfig("example.com", 1024)

Ad Rem

First Approach

So, keeping with the theme, let’s wire up a life support system for plants. Say we have two HTTP services - one for moisture control, and the other handling miscellaneous sensors:

services.scala
class HydroService(val config: ServerConfig) {
    //logic goes here
}

class SensorService(val config: ServerConfig) {
   //logic goes here
}

where, again, the config class is:

case class ServerConfig(host: String, port: Int)

So, our config file could look like this:

app {
  hydro {
    host: "hydro.example.com"
    port: 8215
  }

  sensors {
    host: "sensors.example.com"
    port: 1777
  }
}

To make things sweet and simple, let’s encapsulate the the entire (relevant) config into a container class:

AppConfig.scala
class AppConfig(val hydro: ServerConfig, val sensors: ServerConfig)

case class ServerConfig(host: String, port: Int)

We’ll now take the first stab at the application runner:

Run.scala
import com.softwaremill.macwire._
import com.typesafe.config.ConfigFactory

object Run extends App {

  lazy val rawConfig = ConfigFactory.load()

  import net.ceedubs.ficus.Ficus._
  import net.ceedubs.ficus.readers.ArbitraryTypeReader._

  lazy val config: AppConfig = rawConfig.as[AppConfig]("app") (1)

  import config._ (2)

  lazy val hydroService = wire[HydroService]
  lazy val sensorService = wire[SensorService]

  println(s"HydroService configured with ${hydroService.config}") (3)
  println(s"SensorService configured with ${sensorService.config}")

}
1 It is a good idea for your app’s config to be contained in a dedicated subpath, mostly for reasons of avoiding namespace clashes, but also due to technical difficulties of loading the "root path".
2 Bringing ServerConfig instances into scope.
3 Debug statements.

Obviously, we’ll end up with:

> runMain Run
[info] Updating {file:}root-201604ficusplay...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 3 Scala sources to target/scala-2.11/classes...
[error] src/main/scala/Run.scala:24: Found multiple values of type [ServerConfig]: [List(sensors, hydro)]
[error]   lazy val hydroService = wire[HydroService]
[error]                               ^
[error] src/main/scala/Run.scala:25: Found multiple values of type [ServerConfig]: [List(sensors, hydro)]
[error]   lazy val sensorService = wire[SensorService]
[error]                                ^
[error] two errors found

because Macwire cannot distinguish between the two ServerConfig instances pulled into scope with import config._.

So, what can we do?

Tagging to the rescue

Well, it turns out Macwire possesses the notion of Qualifiers. In intent, they are identical to JSR 330 qualifiers, the only difference is that they operate on types instead of annotations (since Macwire performs its DI based on types).

OK, so we need to qualify our two service dependencies with some sort of tagging types. We could create marker traits for that. However, we notice that we:

  • already have two distinct types

  • that unambiguously convey our intent.

I’m talking, obviously, about the service types themselves!

OK, so let’s tag those config dependencies:

services.scala
import com.softwaremill.tagging._

class HydroService(val config: ServerConfig @@ HydroService)

class SensorService(val config: ServerConfig @@ SensorService)

Of course, for MacWire to know where to get these dependencies from, we have to tag the config class as well:

AppConfig.scala
import com.softwaremill.tagging._

class AppConfig(val hydro: ServerConfig @@ HydroService, val sensors: ServerConfig @@ SensorService)

case class ServerConfig(host: String, port: Int)

OK, looks like we’re all set. Let’s run our app now, aaand:

[info] Compiling 1 Scala source to target/scala-2.11/classes...
[error] src/main/scala/Run.scala:20: Cannot generate a config value reader for type
com.softwaremill.tagging.@@[ServerConfig,HydroService], because value readers cannot be auto-generated for types with type parameters.
Consider defining your own ValueReader[com.softwaremill.tagging.@@[ServerConfig,HydroService]]
[error]   lazy val config: AppConfig = rawConfig.as[AppConfig]("app")

Still no go.

Rescuing the tagging

The error message pretty much spells out the problem [1]. Since the config instances are now of type @@[ServerConfig,XService], Ficus is unable to find a way to construct instances of their types.

Via the error message, we’re offered a solution of implementing a ValueReader, which is what Ficus depends on when transforming config files into instances. Fortunately, from our previous attempt, we know that Ficus already has ValueReader objects in scope capable of generating case classes like ServerConfig [2]. Additionally:

  • ValueReader has a map method,

  • MacWire provides a taggedWith[TTag] helper that converts any type TType into @@[TType, TTag].

Let’s put those pieces together and add our custom reader for tagged types:

implicit def taggedReader[TType, TTag](implicit reader: ValueReader[TType]) = reader.map(_.taggedWith[TTag])

After we add the above to run, we should end up with:

> runMain Run
[info] Compiling 1 Scala source to target/scala-2.11/classes...
[info] Running Run
HydroService configured with ServerConfig(hydro.example.com,8215)
SensorService configured with ServerConfig(sensors.example.com,1777)
[success] Total time: 2 s, completed 2016-04-30 15:08:12

And we’re pretty much done!

One final improvement that we can make is to better convey the relationship between the transformed type and ValueReader. We do this by using the following equivalent form of our reader:

implicit def taggedReader[TType: ValueReader, TTag] = implicitly[ValueReader[TType]].map(_.taggedWith[TTag])

In the end, our app class looks like this:

Run.scala
import com.softwaremill.macwire._
import com.softwaremill.tagging._
import com.typesafe.config.ConfigFactory
import net.ceedubs.ficus.readers.ValueReader

object Run extends App {

  lazy val rawConfig = ConfigFactory.load()

  import net.ceedubs.ficus.Ficus._
  import net.ceedubs.ficus.readers.ArbitraryTypeReader._


  implicit def taggedReader[TType: ValueReader, TTag] = implicitly[ValueReader[TType]].map(_.taggedWith[TTag])

  lazy val config: AppConfig = rawConfig.as[AppConfig]("app")

  import config._

  lazy val hydroService = wire[HydroService]
  lazy val sensorService = wire[SensorService]

  println(s"HydroService configured with ${hydroService.config}")
  println(s"SensorService configured with ${sensorService.config}")

}

Summary

We’ve managed to get Macwire and Ficus happily working together to parse HOCON config files, in a type-safe manner.

Note that creating a master config object in larger, multi-module projects, is a Bad Idea™. However, that’s not a problem, since you can just pass on the "raw" config (from typesafe-config) onto the relevant submodules, and have Ficus generate the actual config instances there.

Overall, this approach scales well (architecturally), and requires only the minimal boilerplate necessary for type safety.

You can find the full code example on GitHub.

1. If only more Scala libs took this degree of care with compile-time messages…​
2. Since the original approach went pass the "Ficus stage" successfully, and only emitted an error during wiring.

Learn AND remember with Anki: Part 4: Statistics and Conclusions

Mikołaj Koziarkiewicz

Previously

We’ve just went over a practical example of the scheme. Now, let’s talk stats and conclusions.

So how does it work out, really?

While I can’t offer you any peer-reviewed, double-blind study on the validity of the method I’ve described above. Subjectively however, I can tell you that I am convinced adopting the technique has had a great effect on improving the time effectiveness of learning stuff I want to know. Without fail, for every such source, whether a book, a course, technical documentation or otherwise, I have progressed beyond just that "feel-good" sensation and actually was able to recall the most important information many, many months after first absorbing it.

However, I do have some objective, if not statistically representative, data. Take a look at the following figures, which describe the total time and total number of card reviews throughout my entire usage history, with my ~850 card collection, grown over time:

anki stats
Stats since day 1. Note the "Average for days studied" entry.

As you can see, the amount of time per day, even if artificially inflated by the aforementioned commuting, is laughably small when compared to the benefits. The benefits themselves, in turn, are indirectly demonstrated by the portion of "relearn" (i.e. "I forgot all about it") card reviews in the overall scheme - Anki/SRS really is that effective in helping you remember [1].

In closing

It would be a disservice to Anki if I’d fail to mention that it is also an excellent tool for helping in learning a language. Anki was, after all, originally created to aid its author in studying Japanese.

In fact, a comprehensive collection of community-created decks has been made available, spanning not only multiple languages but also subjects such as Biology, Geography, Physics, and others.

Finally, I hope I have shared some of the enthusiasm for Spaced-Repetition-assisted learning in the context of technical knowledge [2]. I encourage you to try it out for a small dataset, and see how it goes - as long as you have a minimum of self-control and curiosity, you should benefit from adopting this technique. Maybe it won’t help you win "Jeopardy!", but it will certainly provide you with a powerful tool when dealing with the ever increasing corpus of information that must be absorbed in order to stay up to date, and to keep in touch with the "core" knowledge that serves as the foundation for your daily technical decisions.

1. A recent example - after preparing a simple deck, and some minimal practice, I was able to write complex MongoDB aggregations off the top of my head, many months after completing the related online course.
2. and haven’t bored you to tears while at it.