Reactive Streams in Scala meet a Game Engine - Part 2

16 November 2016

Unfortunately, just after the first installment of this series, I’ve involved myself in another side-project, which will probably be released soon. But hey, that’s no excuse to at least make a small update in the meantime, to keep the ball rolling, right?

Scope

Desiderata

As mentioned in the first post, the next step should be defining our scope. This basically boils down to making our exploration of the streaming paradigm as efficient as practicable, specifically identifying problems intrinsic to game programming.

One of those is efficient resource allocation. Games need to spawn and remove potentially large numbers of object quickly, and without taxing the memory of the device. For JVM-made games, Android device limits would probably serve as the most prominent example in recent years[1]. Game engines like libgdx mitigate this problem by employing pools of mutable data structures (which reduces the number of created objects, and hence GC time), among other tricks.

Another issue is smooth rendering of the game’s updates. Enjoyment drops sharply if the game becomes visibly choppy.

A related problem is responsiveness to player input. Smooth FPS gives little consolation if the player remains constantly frustrated while unable to react in time to the unfolding events.

To sum up, we would like to have a game that:

  • requires relatively quick player reaction (so e.g. no turn-based games),

  • has a lot of events happening at any given moment,

  • and employs a large number of entities to do so.

First sketches

Obviously, we want some kind of an arcade game, somewhat close in style to a shoot’em up, even drifting into bullet hell territory[2].

For starters, we want some sort of controllable character, multiple, respawning enemies, and some sort of score meter (which will help us gauge the rate of updates).

Here’s a veeery rudimentary sketch of how the game will look like at the beginning:

starter sketch
Figure 1. No, I’m not a graphic artist, why do you ask?

Of course, we’ll keep adding stuff as we go along, but we already have more than enough work to do.

Up Next

In the following episode we’ll finally see some code. We’re going to define the boilerplate skeleton of our application, and create an Akka Streams source that allows to react to game state updates.


1. Alongside Minecraft’s chunk management.
2. Apologies for linking to TV Tropes.


Reactive Streams in Scala meet a Game Engine - Part 1

12 September 2016

Intro

It’s always interesting to see a new (or newly fashionable) technique being applied to new and unexpected areas, and see whether it makes sense (and what makes or breaks it).

Such is the case with Scala (reactive) streaming and computer game creation. A while ago, I stumbled upon a talk by Aleksandar Prokopec, which planted the seed of inspiration for this blog series. And what set that seed on a path of growth was another presentation, this time by Michał Płachta [1].

So, why another thing on this?

Well, of course it wouldn’t be fun if the blog series was just a same-scenario rehash. Here’s what’s going to be different - while the aforementioned talks were more-or-less "from-scratch" affairs, creating a game engine and/or a streaming platform along the way - I will be going the "lazy" route, and attempting to join together a ready-made game engine with a ready-made streaming framework.

The primary advantage of this kind-of-unique[2] setup is that the focus almost by definition falls on the paradigm mismatch problem, and thus it’s easy to see what one must solve to make the whole shebang work.

And there’s a related blessing in disguise - because we’re operating between two rigid boundaries of stable libraries, any domain-related idiosyncrasies will be easily identifiable, and have a reduced chance of rising up later on to bite is in the hind parts.

Now, let’s talk framework choices.

Decisions, decisions

Game Engine

It may come as a surprise to some of the readers, but JVM game programming is not exactly a barren field, despite the infamous "Java is slow" stereotype. Probably the greatest influence in recent years has been the rise of Android, hence why a high proportion of JVM game engines provide support for both "mobile" and "desktop" targets[3].

In any case, two engines seem to be the strongest contenders at the moment: libGDX and jMonkeyEngine. Both have been present on the scene for several years, and offer comprehensive game development feature sets[4].

The choice, however, falls on libGDX, for two reasons: it has explicit support for 2D rendering, and it’s the one I’m actually familiar with.

I’m going to expand on libGDX in upcoming installments. Right now, let’s roll over to the other side for the…​

Streaming Implementation

Apart from the obvious choice of Akka Streams, I was also monitoring the progress of Swave.

I recommend following its development closely. It is, in implementation and design, significantly less complex than Akka Streams. Swave can therefore serve as a convenient stepping stone in beating the learning curve, when adapting to the streaming paradigm.

However, Swave is still shaping up, its documentation is somewhat lacking[5], and I have a sneaking suspicion that some unique features of its "competitor" (like blueprinting) might come in handy, so we’re sticking with Akka Streams.

OK, so the tech base is decided now. All that remains is one tiny detail - just what is the end goal and how to achieve it?

Some ground rules

Let’s start with the goal itself, since it’s pretty obvious - a natural way to verify whether something works (or doesn’t), is through a practical proof-of-concept. Therefore, during this series, I’m going to work towards a fully-playable simple game, that uses streaming as its core operational model.

This leads nicely to the preferred form of the series, i.e. a development blog - meaning:

  • installments will be posted (hopefully) regularly,

  • with a soft word-length cutoff,

  • mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions[6].

Finally, to spice things up a bit, I’m going to deliberately forgo gathering "technical" inspiration from the aforementioned related talks and projects (like the additional abstractions introduced by Dr. Prokopec), in order to see where the problem domain lends toward convergent solution creation.

Up Next

In the next installment, we’re going to cover defining the scope of the PoC game. Stay tuned!


1. Source code here.
2. There are at least two similar projects. One went in the direction of providing an Rx* translation layer. It seems to be inactive for about two years now. Another, implemented in Kotlin, seems to be under active development. Both use the Rx* family of libraries.
3. Be aware 'though that since Android supports "native applications", a lot of Android games are written in C/C++ instead.
4. To answer a - perhaps - lingering question: Minecraft, the poster child of JVM games, uses neither. It has a proprietary engine, with several native bridges for OpenGL support etc.
5. Do see the presentations 'though for a good feel of what it offers.
6. Also making for a nice excuse in case of eggregious blunders.


Painless, type-safe config file parsing with Macwire and Ficus

30 April 2016

Intro

Scala has a distinct advantage of an "official" configuration file format, i.e. HOCON. Even better, the niceness transcends the officialdom - HOCON is quite well-defined, and it has lots of useful of features. In other words, it’s simply a good format.

Of course, any Scala developer past their Day 3 can tell you of a certain little quirk - the reference parser implementation is a Java library. Even though Java-Scala interop is actually mostly tolerable once one learns the ropes, it still remains slightly annoying.

This post will describe a possible approach for reducing that annoyance by synergizing two well-known libs.

Note
The following is intended for devs already minimally familiar with Macwire. If you’re not, take a look at the README. Reading the the Introduction will suffice.

Config

There are several popular pure-Scala HOCON parsing libraries available. We’re going to use Ficus.

As a library, it’s extremely straightforward - specify the config file, the subpath within it, the type you want it to be parsed into - the lib will do the rest. It even supports extraction into semi-arbitrary data types, like so:

server {
    host: "example.com"
    port: 1024
  }
case class ServerConfig(host: String, port: Int)

import net.ceedubs.ficus.Ficus._
import net.ceedubs.ficus.readers.ArbitraryTypeReader._

ConfigFactory.load().as[ServerConfig]("server")
//emits ServerConfig("example.com", 1024)

Ad Rem

First Approach

So, keeping with the theme, let’s wire up a life support system for plants. Say we have two HTTP services - one for moisture control, and the other handling miscellaneous sensors:

services.scala
class HydroService(val config: ServerConfig) {
    //logic goes here
}

class SensorService(val config: ServerConfig) {
   //logic goes here
}

where, again, the config class is:

case class ServerConfig(host: String, port: Int)

So, our config file could look like this:

app {
  hydro {
    host: "hydro.example.com"
    port: 8215
  }

  sensors {
    host: "sensors.example.com"
    port: 1777
  }
}

To make things sweet and simple, let’s encapsulate the the entire (relevant) config into a container class:

AppConfig.scala
class AppConfig(val hydro: ServerConfig, val sensors: ServerConfig)

case class ServerConfig(host: String, port: Int)

We’ll now take the first stab at the application runner:

Run.scala
import com.softwaremill.macwire._
import com.typesafe.config.ConfigFactory

object Run extends App {

  lazy val rawConfig = ConfigFactory.load()

  import net.ceedubs.ficus.Ficus._
  import net.ceedubs.ficus.readers.ArbitraryTypeReader._

  lazy val config: AppConfig = rawConfig.as[AppConfig]("app") (1)

  import config._ (2)

  lazy val hydroService = wire[HydroService]
  lazy val sensorService = wire[SensorService]

  println(s"HydroService configured with ${hydroService.config}") (3)
  println(s"SensorService configured with ${sensorService.config}")

}
  1. It is a good idea for your app’s config to be contained in a dedicated subpath, mostly for reasons of avoiding namespace clashes, but also due to technical difficulties of loading the "root path".

  2. Bringing ServerConfig instances into scope.

  3. Debug statements.

Obviously, we’ll end up with:

> runMain Run
[info] Updating {file:}root-201604ficusplay...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 3 Scala sources to target/scala-2.11/classes...
[error] src/main/scala/Run.scala:24: Found multiple values of type [ServerConfig]: [List(sensors, hydro)]
[error]   lazy val hydroService = wire[HydroService]
[error]                               ^
[error] src/main/scala/Run.scala:25: Found multiple values of type [ServerConfig]: [List(sensors, hydro)]
[error]   lazy val sensorService = wire[SensorService]
[error]                                ^
[error] two errors found

because Macwire cannot distinguish between the two ServerConfig instances pulled into scope with import config._.

So, what can we do?

Tagging to the rescue

Well, it turns out Macwire possesses the notion of Qualifiers. In intent, they are identical to JSR 330 qualifiers, the only difference is that they operate on types instead of annotations (since Macwire performs its DI based on types).

OK, so we need to qualify our two service dependencies with some sort of tagging types. We could create marker traits for that. However, we notice that we:

  • already have two distinct types

  • that unambiguously convey our intent.

I’m talking, obviously, about the service types themselves!

OK, so let’s tag those config dependencies:

services.scala
import com.softwaremill.tagging._

class HydroService(val config: ServerConfig @@ HydroService)

class SensorService(val config: ServerConfig @@ SensorService)

Of course, for MacWire to know where to get these dependencies from, we have to tag the config class as well:

AppConfig.scala
import com.softwaremill.tagging._

class AppConfig(val hydro: ServerConfig @@ HydroService, val sensors: ServerConfig @@ SensorService)

case class ServerConfig(host: String, port: Int)

OK, looks like we’re all set. Let’s run our app now, aaand:

[info] Compiling 1 Scala source to target/scala-2.11/classes...
[error] src/main/scala/Run.scala:20: Cannot generate a config value reader for type
com.softwaremill.tagging.@@[ServerConfig,HydroService], because value readers cannot be auto-generated for types with type parameters.
Consider defining your own ValueReader[com.softwaremill.tagging.@@[ServerConfig,HydroService]]
[error]   lazy val config: AppConfig = rawConfig.as[AppConfig]("app")

Still no go.

Rescuing the tagging

The error message pretty much spells out the problem [1]. Since the config instances are now of type @@[ServerConfig,XService], Ficus is unable to find a way to construct instances of their types.

Via the error message, we’re offered a solution of implementing a ValueReader, which is what Ficus depends on when transforming config files into instances. Fortunately, from our previous attempt, we know that Ficus already has ValueReader objects in scope capable of generating case classes like ServerConfig [2]. Additionally:

  • ValueReader has a map method,

  • MacWire provides a taggedWith[TTag] helper that converts any type TType into @@[TType, TTag].

Let’s put those pieces together and add our custom reader for tagged types:

implicit def taggedReader[TType, TTag](implicit reader: ValueReader[TType]) = reader.map(_.taggedWith[TTag])

After we add the above to run, we should end up with:

> runMain Run
[info] Compiling 1 Scala source to target/scala-2.11/classes...
[info] Running Run
HydroService configured with ServerConfig(hydro.example.com,8215)
SensorService configured with ServerConfig(sensors.example.com,1777)
[success] Total time: 2 s, completed 2016-04-30 15:08:12

And we’re pretty much done!

One final improvement that we can make is to better convey the relationship between the transformed type and ValueReader. We do this by using the following equivalent form of our reader:

implicit def taggedReader[TType: ValueReader, TTag] = implicitly[ValueReader[TType]].map(_.taggedWith[TTag])

In the end, our app class looks like this:

Run.scala
import com.softwaremill.macwire._
import com.softwaremill.tagging._
import com.typesafe.config.ConfigFactory
import net.ceedubs.ficus.readers.ValueReader

object Run extends App {

  lazy val rawConfig = ConfigFactory.load()

  import net.ceedubs.ficus.Ficus._
  import net.ceedubs.ficus.readers.ArbitraryTypeReader._


  implicit def taggedReader[TType: ValueReader, TTag] = implicitly[ValueReader[TType]].map(_.taggedWith[TTag])

  lazy val config: AppConfig = rawConfig.as[AppConfig]("app")

  import config._

  lazy val hydroService = wire[HydroService]
  lazy val sensorService = wire[SensorService]

  println(s"HydroService configured with ${hydroService.config}")
  println(s"SensorService configured with ${sensorService.config}")

}

Summary

We’ve managed to get Macwire and Ficus happily working together to parse HOCON config files, in a type-safe manner.

Note that creating a master config object in larger, multi-module projects, is a Bad Idea™. However, that’s not a problem, since you can just pass on the "raw" config (from typesafe-config) onto the relevant submodules, and have Ficus generate the actual config instances there.

Overall, this approach scales well (architecturally), and requires only the minimal boilerplate necessary for type safety.

You can find the full code example on GitHub.


1. If only more Scala libs took this degree of care with compile-time messages…​
2. Since the original approach went pass the "Ficus stage" successfully, and only emitted an error during wiring.


Learn AND remember with Anki: Part 4: Statistics and Conclusions

26 August 2015

Previously

We’ve just went over a practical example of the scheme. Now, let’s talk stats and conclusions.

So how does it work out, really?

While I can’t offer you any peer-reviewed, double-blind study on the validity of the method I’ve described above. Subjectively however, I can tell you that I am convinced adopting the technique has had a great effect on improving the time effectiveness of learning stuff I want to know. Without fail, for every such source, whether a book, a course, technical documentation or otherwise, I have progressed beyond just that "feel-good" sensation and actually was able to recall the most important information many, many months after first absorbing it.

However, I do have some objective, if not statistically representative, data. Take a look at the following figures, which describe the total time and total number of card reviews throughout my entire usage history, with my ~850 card collection, grown over time:

anki stats
Stats since day 1. Note the "Average for days studied" entry.

As you can see, the amount of time per day, even if artificially inflated by the aforementioned commuting, is laughably small when compared to the benefits. The benefits themselves, in turn, are indirectly demonstrated by the portion of "relearn" (i.e. "I forgot all about it") card reviews in the overall scheme - Anki/SRS really is that effective in helping you remember [1].

In closing

It would be a disservice to Anki if I’d fail to mention that it is also an excellent tool for helping in learning a language. Anki was, after all, originally created to aid its author in studying Japanese.

In fact, a comprehensive collection of community-created decks has been made available, spanning not only multiple languages but also subjects such as Biology, Geography, Physics, and others.

Finally, I hope I have shared some of the enthusiasm for Spaced-Repetition-assisted learning in the context of technical knowledge [2]. I encourage you to try it out for a small dataset, and see how it goes - as long as you have a minimum of self-control and curiosity, you should benefit from adopting this technique. Maybe it won’t help you win "Jeopardy!", but it will certainly provide you with a powerful tool when dealing with the ever increasing corpus of information that must be absorbed in order to stay up to date, and to keep in touch with the "core" knowledge that serves as the foundation for your daily technical decisions.


1. A recent example - after preparing a simple deck, and some minimal practice, I was able to write complex MongoDB aggregations off the top of my head, many months after completing the related online course.
2. and haven’t bored you to tears while at it.


Learn AND remember with Anki: Part 3: From theory to practice

04 August 2015

Previously

During the last episode, we’ve discussed a learning scheme that can help you to avoid having most acquired information from floating away.

In what follows, we’ll use an example to walk through the scheme step by step.

Intended audience

While the bulk of this series offers information that should be useful for virtually everyone, the examples below are tailored towards programmers, specifically JVM polyglot programmers. You have been warned.

Enough theory, time for practice

Step 0: Choose what to learn

For this article, I’ve decided to use the example of ScalaTest matchers. That topic makes a good example, because:

  • the subjects squarely in the category of "in one ear, out the other" ,

  • remembering brings a tangible benefit - reports for the specialised matchers are more meaningful than those from a simple should be equal.

Let’s start, shall we?

Step 1: Highlight Key Passages (Optional)

The matcher documentation (again, available here) is actually a good case where this step can be skipped. The docs are compact, information-rich, and the most notable facts are plainly visible.

Step 2: Create Mind-map-like notes

Before we start, if you don’t have a favorite mind-mapping application, download and install Freemind.

First of all, we’re not actually going to do the full docs, just enough to create a relatively good example. So, let’s do the first 3 sections, up to and including "checking strings".

Here’s how a note-centered mind map might look like:

freemind scalatest
The resultant mind map.

You might notice the following things:

  • consistently with "normal" mind maps, items progressively become less general and more specific the further away from the center,

  • also like "normal" mind maps, items connected "physically" are also related semantically [1],

  • unlike typical mind maps, there very little formatting and decorations were applied. This is because the map is intended to be a transitory step, and not something you will revisit often. Normally, the only formatting I do is hyperlinks and icons for denoting especially important stuff.

  • only a portion of the information from the doc page is included. It’s a deliberate decision - the goal is to provide a source for quickly writing up cards, which are supposed to help you in the long-term - so we’re looking for stuff that’s most useful. In other words, we need the info for your regular life/work, and not to prepare for a fire-and-forget exam.

Step 3: Extract flashcards from the mind map in Anki Desktop

If you haven’t done so already, download and install Anki on your computer.

First, click Create Deck [2]. Since I already have a Scala deck, I’m going to make a subdeck, which is done by writing Scala::ScalaTest. After the deck is created, click on it.

anki newdeck
Ready to add cards.

Now, we’re ready to create the cards proper. Click on Add at the top [3].

anki newcard
New card to be filled out.

You’ll be presented with a view similar to the above (the additional UI elements mostly come from the source code plugin).

We’re going to out of order and start with the second point - asserting size and/or length - since it’s more straightforward. Let’s start with the length:

  • Front: I recommend this to be titled in the way that is the most similar to how you’d try to recall the relevant information. In this case, I’d write "ScalaTest - asserting length".

  • Back: this should contain the relevant information in the most concise way. In this case, all we need is the text of the assert, e.g. have length X [4].

anki newcard filled
Card filled out.

Now, do the same for asserting size. Here you might think whether it wouldn’t be more efficient to store both pieces of information in a single card. In practice, I find that it’s much more worthwhile to split information into as many cards as possible, as simple cards are much easier to learn and hence produce a better retention effect.

Going up next, we have the string comparisons. Here, I’d suggest to create 8 cards:

  • 3 per each "simple" matcher

  • 4 per each "regex" matcher

  • 1 for the withGroups matcher qualifier.

anki card simple filled
Simple comparison, note slogan-like front naming.

After that, we’ve got comparisons. Here, I’m going to create just two cards, one per every strict/equal-or pair, since the cases appear to be both semantically and "graphically" coupled.

anki card standard
Two pieces of info, but still pretty straightforward.

Note that here I’m omitting the Ordering[T] information, as I’m making the assumption that I can rely on the compiler for recalling that tidbit. However, if you e.g. use lots of custom classes with dedicated Ordering implementations, creating an additional card might be worth your while.

Finally, let’s go back to the general equality case. The be card is pretty straightforward:

anki card complex entry
Simple equality.

However, for the equal we have to cram some info on the back:

anki card complex
The most complex card type you should be making.

This is unfortunate, however sometimes you have pieces of information that are inseparable.

And there you have it! If you want to study using the cards created for this section, download the pack from here.

Step 4: Learn from flashcards in the Anki Mobile Client(s)

Now that we’ve created the cards, we can move to actually using them. For that, you’ll need to:

  1. Get the mobile client, in either the Android or iOS versions

  2. Set up AnkiWeb synchronization between your desktop client and your mobile device.

Important
Please be aware of the content policies before you sync data through AnkiWeb.

Now that you’ve set it up, and open the synced deck, you should see something like the following:

anki mobile starting
Before…​
anki mobile starting answer
…​and after the first answer.

You’re viewing a new card in the learning stage, which will require at least one repetition on the same day. Always choose the option that authentically reflects your recall ability of that card. Eventually, you should arrive at something similar to the following screen:

anki mobile final review
Last answer for now. Note the different time periods.

Now the choices extend beyond the same day. If your recall is increased effectively, the interval selection will lengthen at a geometric rate. So when in doubt, answer pessimistically - it won’t "cost" you much time.

Use the Widget

If you’re using Android, the widget is a nice feature. It shows you the amount of pending cards for the day, as well as the projected time required for going over them.

Step 5: Review, Heal, and Grow your flashcard "Deck"

This is probably the most personalized of the steps. However, a couple common issues might manifest, most notably:

Errors on the cards

Factual, typographical or otherwise: minor ones can be remedied immediately through the mobile client, but for more complex stuff, your best bet is to mark the card by favoriting it, and potentially burying the card, "postponing" it to the following day, if the problem is severe enough so that you can’t meaningfully review the card [5].

anki bury favorite
Hide/Delete will show Bury.

Lack of perceived benefit from using the cards for a particular topic

This mostly stems from not enough understanding of the subject matter. With this method, always try to make a conscious effort to actually comprehend and integrate the source material. This even applies to our relatively simple example - the retention effect for the ScalaTest will be much greater if you practice writing the matcher expressions during the the initial days.

General frustration

In other words, the bog standard reaction to a new habit that one is trying to form. After the initial enthusiasm dwindles, the act of repeating the cards might appear like a chore. In this case, I advise you to

  • simply stick it out if you’re only a couple of weeks in,

  • be honest - if you forgot the card’s content, choose "Again",

  • otherwise try to remember situations where the use of this technique has improve recall, and

  • take note of how much time have you’ve already spent by using this method [6].

Recalling knowledge you no longer need

It happens, whether when studying for actual exams, investing in learning a technology that didn’t pan out, or for other reasons. In this case, you just need to suspend the relevant cards, causing them to no longer appear until they’re manually resumed.

Amassing a huge backlog

  • To prevent that, start small (<100 cards total) until you get the gist of it, and try to set aside some regularly scheduled period in your daily routine for the card review - for example, during your daily commute [7].

  • To resolve it, just chip away at the mass of cards for several days. Due to how the SRS algorithm works, you are bound to eventually reduce the backlog to a manageable size.

Coming Up

In the final part of the series, we’ll look at some actual statistics related to using Anki, and touch on some miscellaneous closing points.


1. There are no additional connections in this particular example, but, in general, it’s OK to make them.
2. A deck is simply a collection of Anki cards.
3. Yes, the Anki UI is sometimes all over the place.
4. Pardon the faint red markings on the illustration - there seems to be a bug in the Anki Linux client that makes disabling the spellcheck impossible, hence a quick photoedit job.
5. Often happens with formulas encoded in LaTeX, as the mobile client is notoriously fickle when displaying them.
6. Yes, I’m basically suggesting an abuse of humanity’s susceptibility to the Gambler’s Fallacy.
7. Which was how I started.


Learn AND remember with Anki: Part 2: Putting it all together

23 June 2015

Previously

In the starting article, I’ve provided a quick overview of what Spaced Repetition means, and talked about the tools that can facilitate it. Now, let’s see how we can put these tools to good use.

The Scheme

Based on using Anki for knowledge retention, I’ve developed a learning scheme to keep myself from forgetting the stuff I’d rather remember. Given a new source material (e.g. a book, an online course), here’s how it looks like.

Step 0: Choose what to learn

Probably the most important step - you need to consciously, and responsibly, ask yourself the question: "what knowledge do I need to retain effectively"? This is something that you optimally should work out for yourself.

However, I can provide three pointers:

  • The best fit should align with one of the two categories I’ve described in the introductory sections.

  • You already need to understand whatever you’re trying to recall - the scheme I’m describing want help you with that.

  • Don’t start with adding "feel-good stuff" - knowledge that you always wanted to recall off the top of your head in order to impress yourself. You’ll most likely, eventually get frustrated and see the exercise as a waste of time. Do begin with applying this learning method to things you actually need.

Step 1: Highlight Key Passages (Optional)

This is of course only relevant for textual sources. I still recommend it whenever you’re learning from those, because it helps you stay in the flow.

Step 2: Create Mind-map-like notes

Regularly, for each piece of content (video for online courses, chapter for a book, etc.), add notes to a mind map describing the thing you’re currently learning about [1].

This may seem like a superfluous step, but it helps in three things:

  • it actually helps with knowledge retention by preliminarily organizing it,

  • it makes the next step go faster, since you don’t have to re-read/-watch the entire content,

  • most importantly, it helps to verify whether you’ve understood what you’re learning about. You will very likely to see that you’re e.g. writing nonsense at this point, and it’s less frustrating to identify it and fix it than in the next step.

Note
I say "mind-map-like" because someone might argue that the notes might not necessarily look like a "proper" mind map, i.e. there are no colors, graphics, etc. Nevertheless, I think the form of a tree, which mindmaps take, is the most conducive to general, rapid note-taking.

For the actual mind mapping I prefer to use Freemind. It’s open source, free, multiplatform, and offers a set of convenient keyboard shortcuts for quick note-taking.

freemind example
A mind-map example - an (outdated) mind map of Akka documentation.

Step 3: Extract flashcards from the mind map in Anki Desktop

Now that you have rapid-access notes, it’s time to go through them and create the flashcards. We’ll mostly be dealing with "classic cards", i.e. with two faces - an entry, and whatever you should recall according regarding the entry. However, there are also other types: reversed (where the "entry"-"recall" relation is bilateral, effectively doubling the given card), and cloze completion, where you "fill" in text (useful for e.g. learning phrases in foreign languages).

Also, you can insert almost everything into a card, including:

  • plain and formatted texts,

  • pictures,

  • sounds,

  • LaTeX, including complex formulas in math mode,

  • source code, with the correct plugin [2].

See the figure in the introductory section for some examples.

Step 4: Learn from flashcards in the Anki Mobile Client(s)

anki learning example
Reviewing a card in Anki.

Whereas the Desktop client is more conducive to creating (and reviewing) cards, I find it much better to actually learn by using the Anki software on mobile devices (iOS/Android [3]).

Principally, this is because, with the mobile client, you have less distractions. And trust me, during the beginnings, once the initial enthusiasm wears out but before you see tangible benefits, you will need all the focus you can get.

Another, equally important thing is that you will have access to the cards wherever you go. This makes it an easier to form the habit of reviewing the cards daily, especially in the case of the Android client, where you can also add a widget to your home screen.

Step 5: Review, Heal, and Grow your flashcard "Deck"

Your cards, once created, should not be treated as if set in stone. Splitting a card, improving it, suspending it, or removing it altogether should all be options you need to consider.

Coming Up

In the next installment, we’ll go from theory to practice and see how it pans out.


1. I usually keep a map e.g. per book.
2. Did I mention Anki has a large plugin directory?
3. While the Android version is completely Open Source, the iOS version is paid-for, and, in fact, the latter is currently the main source of monetary support from the project.


Learn AND remember with Anki: Part 1: Introduction

16 June 2015

Intro

Have you ever:

  • regretted that you only remember some bare trivia from a course or a book that you studied several months ago? Or maybe even stuff from university/college that you wish you’d recall now?

  • constantly annoy yourself with looking up some implementation detail, API definition or another fact, that you need every couple of weeks, but juuuust manage to slip your mind the next time you require it?

  • feel like learning new stuff (perhaps to progress beyond your current job) is a Sisyphean task that doesn’t net you anything?

If so, than this series of articles is for you.

The Perils of Ad-Hoc Learning

The nice thing about living now is having slightly more free time for your basic non-survival needs. The absolutely scary thing, in turn, is that the amount of information you have access to, and can gain, is simply overwhelming. Understandably this becomes even worse for IT professionals, what with a new library/framework coming out every week [1].

Of course, a lot of this information is fire-and-forget. Another portion you use day to day and, by this virtue, remember without any problems.

What’s left can be divided into two broad categories.

One is knowledge that you use in irregular, but relatively frequent intervals. An example would be an API quirk that you keep re-reading about every couple of weeks. Irritating, isn’t it? Also wasteful.

The other contains "core" knowledge - stuff that you’re not necessarily using directly, but nevertheless benefit from recalling it readily. Forgetting this kind of information is much more insidious - you just end up doing stuff less effectively; or, perhaps one day you suddenly realize that you completely forgot all the things you wanted to learn from the course you did half a year ago.

But, but, the Internet!

A commonly raised counterargument against investing time in recall methods boils down to:

Why bother? Everything is on the Internet anyway, if I don’t remember something, I’ll just look it up!

And sure, you can do that. But how much does it cost you?

Here’s an example: remind yourself of the last small fact or info tidbit that you required and you needed to search for. Now, take your favorite stopwatch app [2], and try out finding the answer on the Internet.

I’d be willing to bet that for most of you, the total time between switching away from this article, and returning to it after searching will be slightly north of 10 seconds.

That’s not a lot, right? But those search bits add up during the day, and you need to factor in the context switch costs [3].

Now try to recall the answer again, from memory. You, quite likely, still can, and it probably took you less than a second to remember. That is an order of magnitude of a difference. Making a somewhat strained analogy, it’s quite similar to the difference between reading from disk and reading from RAM.

Of course, you’re still spending time by putting additional effort into knowledge retention - but that’s dedicated time, as opposed to hacking up your work routine into disjointed bits.

Spaced Repetition - a solution

Obviously no one is going to sit around every day working on remember their cumulatively growing knowledge by rote.

But there are shortcuts to do something very similar much more efficiently.

The concept of Spaced Repetition offers one such shortcut. There are many implementations exploiting the idea, but they all boil down to taking advantage of particularities of the human brain in order to achieve effective recall of various facts and concepts, with relatively minimal effort.

An SR-based approach

One such implementation is Anki. It’s flashcard-based spaced repetition software. Its documentation can be found here, and it’s available for download on:

The tl;dr version of the process of working with flashcards and SRS looks like this:

  • you create a flashcard. In the simplest version it’s a note with two faces: the prompt for what you want to learn, and the thing you want to learn. Here’s a couple of examples.

anki cards overview
A selection of Anki cards: normal text, source code, image, LaTeX formulas.
  • after several initial repetitions, Anki prompts you with the cards, in increasing time intervals: initially a couple of days, then a couple of weeks, months, and so on. At each repetition, when viewing the answer, you are prompted to choose one of the following options:

    • "Again" which means you forgot about the entry, and need to reset the learning process,

    • "Good" meaning you have good recollection, i.e. the vanilla option,

    • "Hard" implying thatas you’ve sorta learned, but aren’t quite confident, leading to a shorter time to repetition,

    • "Easy" that tells the system to extend the time interval to a greater extent than with "Good".

As you have probably figured it out, the biggest benefit from this system is that the time intervals are managed automatically, and you get automatic reminders to repeat your cards (which is especially useful if you install the mobile client).

Local Flavor

Anki’s repetition selection algorithm is a derivative of SM2, which was invented in Poland.

Coming Up

In the second episode of this series, we will talk about a learning scheme developed empirically by Yours Truly, that takes advantage of Anki when acquiring technical (and other) knowledge.


1. and a Javascript implementation following up in 5 days.
2. I expect very few people still have actual stopwatches, digital or analog.
3. Also, let’s not kid ourselves, you probably took this as a challenge and done the search than you would normally do under this kind of situation.


Musings on Ansible

09 April 2015

This is a set of loosely-related remarks and observations, accrued due to recent use of Ansible in several work and personal projects (and on the latter note, I made a thing).

The entry is mostly intended as a writing exercise, so if you’re evaluating Ansible at the moment, please keep this in mind.

A word of Introduction

Ansible, like Chef, Puppet and Salt, is configuration management software. Despite my best intentions to reach the widest audience possible, providing a comprehensive overview of the field would require creating a whole series of blog posts.

Therefore, wanting to convey the following statements in reasonable time, I’m forced to limit the target readership to those that posses basic knowledge of the subject. In other words, if you are unfamiliar with the subject, please do take your time to browse the included links. Sorry!

Ansible - the good, the bad, the ugly

Context

All those remarks are made by someone who is strictly more a dev than an op, so the things I’m writing about here may or may not apply to your situation as much as they did to mine.

Also, the entry was writen when going through the Through of Disillusionment, halfway to the Slope Enlightenment, and therefore may sound more negative than intended.

Good: Workerless setup

The absolutely wonderful thing about Ansible is its agentless architecture - you don’t need to setup anything on the services machines, other than a valid SSH connection to a user (usually with possible sudo access). No special nodes requiring additional setup (like in Chef), nothing of the sort.

This in fact makes it very convenient to bootstrap setups for CI and the like, or even a quasi Inversion-of-Control setup, using the ansible-pull utility.

Such a feature may seem like a small thing, but it reduces the error rate during the "metagame" of setting up your servers.

Good: Easy to understand syntax

Ansible uses YAML with embedded Jinja2 for its configuration definitions, and most of the basic stuff can be expressed that way. Getting the majority of desirable output defined is pretty straightforward, once you learn the basics.

Bad: …​that’s sometimes not as intuitive as it should be

A big stumbling block I’ve encountered is correctly specifying the conditions in when blocks (saying when to execute a task) and similar ones. Truth be told, even after viewing the parsing source code for the "playbooks", I’m still not entirely confident on what is and isn’t allowed.

I think this is due to the fact that the Jinja2-based syntax sits pretty squarely in the Uncanny Valley for someone with an off-and-on Python background. In effect, you end up writing those conditions like in Python, which works…​ about 90% of the time. The remaining 10% will piss you off to no end.[1]

Good: Has a strong focus on idempotency

Pretty much a given for all modern configuration management software, but nevertheless I could excitedly rave and rant about that at you the entire day - it’s awesome that Ansible specifically focuses on what the state should be, rather than what tasks to do.

Good: Trivial to customize

To recap, here’s how config management works in Ansible:

  • the unit of work is a task, meant to encapsulate a single "end-state" quantum, i.e. something that should be ensured to be fulfilled once this task is done.

  • tasks use modules, which do the actual grunt work and can be implemented in most languages (a lot of them use Python, obviously since Ansible is written in that). There exists a cornucopia of Ansible built-in modules, from user management through ensuring a given is in a file, to EC2 instance setup.

  • tasks can be grouped into roles, which can also contain common variables, custom modules etc..

  • task and role mixes are codified into playbooks, which describes what your actual configuration should look like. You "run" the playbooks on your "inventory"[2] to achieve that desired state.

Creating roles

By the way, if you’ve dug through the role documentation and wondered how to create the role scaffolding automatically, here’s how you can do that:

ansible-galaxy init <rolename>

This is of course mentioned only in the Ansible Galaxy documentation, later on.

You’ll notice that such a structure allows you to modularize your configuration management logic as you see fit. That includes creating several "internal" modules for complex actions and sticking them into your roles.[3]

Bad: …​but no code reuse for Python modules

Funilly enough, due to how Ansible is structured, you cannot have "common" code files for Python-based modules.

That sucks, but is probably an edge case for most stuff.

Good: Good introductory documentation

The docs [4] do a very nice job of showing you the ropes. They are very much example-based, and subsequent steps build upon previous knowledge in a logical way.

Ugly: …​with no formal sections

I fail to see, in the documentation, a "Big Picture" overview of how the various components of an Ansible definition are structured.

Apparently, looking at the the latest Stack Overflow poll, I’m in the minority when it comes to education and this may skew my perception, but I would give my right spleen for an EBNF or similarly-formatted formal specification of the Playbook syntax.

Good: A globally shared role repository

Ansible has something called Ansible Galaxy, which is its way to share you common configuration functionality.[5]. It’s pretty easy to use if you just want to find something (the thing I made is also there).

Ugly: …​very nascent in the current state

However, especially if you display perfectionist tendencies [6], you will spend quite a bit of time examining the roles for the functionality your require.

One problem is that, while a rating system exists, it’s severely underused, and you have to fish for the well-made roles.

Another is the presence of very clearly clashing philosophies when designing roles. It’s best demonstrated when comparing ansible roles from Jeff Geerling and the Stouts group.

The former offer a definitely correctly functioning role, but make a metric f-ton of assumption about what a given piece of setup software is going to operate as. The latter, meanwhile, while not being entirely correct idempotency-wise, allow for very diverse configuration variants.

To be honest, I came to prefer the Stouts roles for their ability to set up as I want them over the geerlingguy ones, despite Mr. Geerling’s foray into book writing on Ansible[7].

Overall

The general picture that I’ve painted hopefully shows a framework with a number of nits that you can pick, but built on solid foundations nevertheless. Those solid foundations will provide a payoff as the framework grows and matures, eliminating the smaller problems along the away. Be aware of the shortcomings, but rest assured that I recommend you check out Ansible for you configuration management needs.


1. Of course, for someone working primary in Python, this may not be a problem, due to their probable contact with and prior use of Jinja2.
2. A list of target servers with some labelling and variables.
3. Normally, modules are quite a "big thing", able to be shared stand-alone, but sometimes the convenience of writing in an actual Turing-complete programming language is too great to miss.
4. …​which admittedly took a nearly-non-trivial amount of time for me to fish out out of the sales-pitch-filled landing page…​.
5. If you’ve never used configuration management software, think of it as the equivalent of the Docker Hub Repository.
6. like Yours Truly
7. Can’t comment on the book’s quality, but the roles show good craftsmanship.


Comments added

06 September 2014

Against my better judgment I have added commenting functionality to the site. Hopefully there won’t be much spam - went with IntenseDebate for this reason. It supposedly provides better built-in spam-fighting capability.


Frist post

31 August 2014

Time for an inaugural post. Regarding the present and future contents, there’s more info on the About page.

As you maybe noticed by the header, this page is generated with JBake. It’s also taking advantage of the JBake Gradle plugin by Cédric Champeau, introduced here (incidentally also reminding me to add a Creative Commons license declaration).



Older posts are available in the archive.

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.