Mikołak

Making LLM coding assistants create graphics… and physical objects

Wed, 29 Oct 2025 00:00:00 +0100

Table of Contents

Intro
Warmup: Making infographics
Main course: Prompts to objects
Outro

Illustration: own work^[1] + background based on work of Ezi on Unsplash.

Intro

I’m sure most professional programmers by now have at least a passing familiarity with LLM-based coding assistants, whether through simple chat interfaces, or by trying not to get their prod data deleted by agentic approaches.

They’re not just a gimmick — they provide help with coding, apparent help with coding, and unambiguously deleterious "help" with coding. Key takeaway 💡: LLMs on occasion reduce cognitive load in 🧠 programming and OK, OK: I’ll stop with the oversaturated facsimile of stereotypical LLM writing patterns.

Regardless of what one might think about the current "AI" situation, coding agents, or even code-focused LLM chat assistants do help with the drudgery of programming on occasion.

Here, however, we’ll use both local and cloud-based LLMs to generate various outputs not traditionally thought of as "code": some to increase productivity or provide more options at work, and some to perhaps expand one’s overall toolset.

Before we proceed, one thing must absolutely be noted: for the topics covered, specialized models absolutely do already exist. The point here is not to show what can be done with modern models, but how already-familiar tools can be used in more ways, without any specialized knowledge, or additional deployment requirements.

Warmup: Making infographics

We’ll start with something more universally usable. On occasion, when communicating more complex messages to your coworkers/team, the text grows so rich in content, it becomes easy to lose the general view over the particulars. Which is where a graphical aid would come in handy – such as an infographic. But not everyone has the skills, patience, and/or time to create them, even when taking account the availability of various web tools meant for that purpose.

While it might seem counter-intuitive, generating graphics through a text-to-text coding-focused model is quite easy: just the matter of picking the right format; in our case, Scalable Vector Graphics (SVG).

SVGs are pretty much the most common files for vector graphics – primarily used for diagrams and illustrations – on the Web. The crucial bit about SVGs, however, is that internally, they’re just a completely human-readable XML file.

Here, let’s create a simple infographic for working with an LLM with the following prompt:

Create an SVG file representing a stylized infographic conveying the following information:

One of the primary uses of modern LLMs is coding assistance,

coding assistance supports both:

what’s traditionally seen as "coding languages", such as JavaScript, Java, Python, C/C++ and so on,

but also anything that can be expressed in a code like format: graphics, music, even models of physical objects.

while specialized models are usually better with all of that, being aware of the additional utility of general coding models gives you an advantage without the additional learning curve.

Keep the style clear, but in a typical infographic scheme. Enrich the infographic with small icons illustrating the most pertinent points.

Here’s one result, from Claude Sonnet 4.5 (through Copilot):

This was obtained from just a single prompt, no followups, and looks relatively good!

If you’re curious about the code itself, download the SVG file and open it in a code editor.

Figure 1. A snippet of the resultant SVG file.

Local models don’t fare that well, comparatively, but still produce something approximating the desired result – here’s an example from GLM 4.5 Air after a couple of corrections:

While this has some formatting issues, it at least provides a valid approximation of what we’d want. And the great thing about generating these graphics in format such as SVG, as opposed to your typical genAI model outputting raster images, is that you can easily edit the file and correct any problems.

To wit, this is the same file after ~3 minutes of editing in Inkscape:

Figure 2. Some icons are still mispainted, but at least the layout is significantly improved.

In short, generating SVGs with LLMs conveys a similar advantage as with programming language code: it provides a decent "jump off" point, which then can be changed, refined, and generally made to one’s liking; or just help with writer’s block, especially when not having significant experience with graphic design.

Main course: Prompts to objects

Intro

Now, for our main course: we’ll take the same chat assistants we use for coding, and make them help us create actual physical objects, just from a short concept spec.

Well, OK, for that to happen, we also require a 3D printer. For those unfamiliar with 3D printing: an extremely simplified procedure to go from design to physical object is as follows:

Design the object: typically in a CAD (Computer Aided Design) program, or via an equivalent coding framework (like OpenSCAD); alternatively, use a 3D editor like Blender.
Export the design to a format expressing surface geometry of the object(s), usually STLs nowadays.
Open the STL file in a slicer: a program that converts the objects surface geometry into an instruction sequence for the specific 3D printer setup (almost always split into layers, hence the name "slicer").
Send the instructions to the printer (here, we’re using an FFM printer), monitor the print, especially the first layer, wait for the print to finish, cool down (optional), remove the print, and clean it up as needed.

That first point reveals what we start with: we’re going to prompt the LLMs to create OpenSCAD specifications. Unlike most CAD programs, OpenSCAD is really just a language for specifying 3D models, plus a runtime with relevant functionality (rendering, exports, etc.). While some CAD suites do use human-readable formats, OpenSCAD is the most prominent one actually intended to be directly written and edited by humans.

The plan is as follows: we’ll start with using LLMs to attempt to describe some simple 3D solids, and, depending on the results, proceed to actually practical 3D prints.

Smoke tests

Let’s try a basic scenario to check if the LLMs do encode the ability to actually generate OpenSCAD specifications.

Here’s our starting prompt:

Create an OpenSCAD file that encodes the following:

a cube with a size of 10mm,

a regular triangle with a side of sqrt(2),

a half-sphere of radius 5mm.

We’ll start with cloud models, invoked via Copilot. Using ChatGPT-4.1, we obtain:

Here is an OpenSCAD file that creates the requested shapes. Each shape is positioned so they do not overlap.
// Cube with size 10mm
translate([0, 0, 0])
cube(10);

// Regular triangle with side sqrt(2)
module regular_triangle(side) {
    h = side * sqrt(3) / 2;
    polygon(points=[
        [0, 0],
        [side, 0],
        [side/2, h]
    ]);
}
translate([15, 0, 0])
regular_triangle(sqrt(2));

// Half-sphere of radius 5mm
translate([30, 5, 0])
sphere(r=5, $fn=64);
translate([30, 5, 0])
cube([10, 10, 5], center=true);
difference() {
    translate([30, 5, 0])
    sphere(r=5, $fn=64);
    translate([30, 5, -5])
    cube([10, 10, 10], center=false);
}
The cube is at the origin. The triangle is placed at [15, 0, 0] to avoid overlap. The half-sphere is created using difference() to cut a sphere in half, placed at [30, 5, 0].

Aaand here’s how the result looks like:

Well, we actually got working code, and that’s good… but only the cube looks half-way close to what we intended. To be fair, the triangle is on us, as we specified the parameters for a 3D object as if it was 2D.

Let’s give ChatGPT another go, with the following correction:

The triangle is considerably too small, and the half-sphere renders as a full sphere within a cube. Please fix.

The "full sphere with a block" description actually refers to the presentation before a "cut" operation.

We obtain the following, supposedly corrected code:

Here’s a corrected OpenSCAD file: The triangle now has a more visible size (side length increased to 10mm). The half-sphere is properly rendered by subtracting the lower half of the sphere.
// Cube with size 10mm
translate([0, 0, 0])
cube(10);

// Regular triangle with side 10mm
module regular_triangle(side) {
    h = side * sqrt(3) / 2;
    polygon(points=[
        [0, 0],
        [side, 0],
        [side/2, h]
    ]);
}
translate([15, 0, 0])
regular_triangle(10);

// Half-sphere of radius 5mm
difference() {
    translate([30, 5, 5])
    sphere(r=5, $fn=64);
    // Cut off the bottom half
    translate([25, 0, 5])
    cube([10, 10, 5], center=false);
}
The triangle is now clearly visible. The half-sphere is created by subtracting a cube from the bottom half of the sphere.

resulting in:

Much better, although, with all the hype towards the big-player LLMs, I would have expected for the half-sphere to be at least oriented with the flat face downwards.

Anyway, after asking once more:

Have the triangle extrude upwards with a face height of 10mm.

We can see the following:

Let’s try to form a refined prompt that will get us to our target in one go:

Create an OpenSCAD file that encodes the following:

a cube with a size of 10mm,

an extruded solid with a base consisting of a regular triangle with a side of sqrt(2)*10mm, and an extrusion height of 10mm,

a half-sphere of radius 5mm, with the flat side facing downwards, and located at the origin of the Z axis.

The response is:

Here is an OpenSCAD file that creates the requested shapes, positioned to avoid overlap: Explanation: The cube is placed at the origin. The extruded triangle uses linear_extrude for a 3D solid, with the triangle base sized as specified. The half-sphere is created by subtracting the lower half of a sphere, with the flat face on the XY plane at Z=0.
// Cube with size 10mm
translate([0, 0, 0])
cube(10);

// Extruded regular triangle (side = sqrt(2)*10mm, height = 10mm)
module regular_triangle(side) {
    h = side * sqrt(3) / 2;
    polygon(points=[
        [0, 0],
        [side, 0],
        [side/2, h]
    ]);
}
translate([20, 0, 0])
linear_extrude(height=10)
regular_triangle(sqrt(2)*10);

// Half-sphere of radius 5mm, flat side down at Z=0
difference() {
    translate([40, 5, 5])
    sphere(r=5, $fn=64);
    // Remove lower half
    translate([35, 0, 0])
    cube([10, 10, 5], center=false);
}
This will generate the three shapes as described, each separated along the X axis.

And the code renders to the following:

That looks about like something we wanted.

Baselines

Having established a baseline prompt, let’s run some tests comparing different LLMs. The selection of models stems from both what is available through Copilot, and what is currently thought of as "good" local LLMs tailored for code generation.

The latter was produced through an extremely rigorous and formal approach of an ensemble method composed of the result of a deep research query based on the currently expressed trends in the prosumer space, validated by a cross-reference from the immediately available wetware compute&storage system instance.

In other words: I ran the query on last month’s posts in r/LocalLLaMA and checked it wasn’t BS ,compared to my own impressions.

Local model selection

The models we’re going to use here are all unsloth’s quantizations/optimizations, specifically (name: variant):

Devstral-Small-2507: UD-Q8_K_XL
GLM-4.5-Air: UD-IQ2_XXS
Qwen3-Coder-30B-A3B-Instruct: UD-Q8_K_XL
gpt-oss-20b: F16

As far as configuration goes:

everything runs through llama.cpp (with llama-swap for convenience),
for every model, the context window size is 16k (mostly for a "thinking buffer"),
--jinja (unsloth’s suggested pre-prompts) is enabled wherever recommended in the model card.

A final clarification: the specific choice of quantization originates from the goal of all models fitting in roughly the same amount of VRAM (under 48 GB). This is theoretically not strictly "fair"^[2], but, firstly, it matches actual usage scenarios better, and modern larger models often exhibit surprisingly good performance even with more aggressive quantizations anyway.

Setup

Now, back to our tests. For each model, we’ll do best of three, and show just that.

Some visualizations include solids that are translucent. This is a manual toggle for visualization purposes, i.e. not an artifact of the model output.

Model	Cube	Triangle Extrude	Half-sphere	Notes
GPT-4.1	✅	✅	✅	-
GPT-5 mini	✅	✅	✅	Needed retries.
Claude Sonnet 4.5	✅	✅	✅	Insufficient spacing.
Gemini 2.5 Pro	✅	✅	✅	Better presentation, clean code, included smoothing.
Gemini 2.0 Flash	✅	❌	❌	Insufficient spacing.
Qwen3-Coder-30B-A3B	✅	✅	❌	The sphere is cut, but at the wrong height. Insufficient spacing.
GLM-4.5-Air	✅	✅	✅	Strong overlap.
Devstral Small 2507	✅	✅	❌	-
GPT-OSS 20B	✅	✅	❌	The half-sphere cut is present, but incorrectly offset.

Model

Cube

Triangle Extrude

Half-sphere

Visualization

Notes

GPT-4.1

✅

GPT-5 mini

✅

Needed retries.

Claude Sonnet 4.5

✅

Insufficient spacing.

Gemini 2.5 Pro

✅

Better presentation, clean code, included smoothing.

Gemini 2.0 Flash

✅

❌

Insufficient spacing.

Qwen3-Coder-30B-A3B

✅

❌

The sphere is cut, but at the wrong height. Insufficient spacing.

GLM-4.5-Air

✅

Strong overlap.

Devstral Small 2507

✅

❌

GPT-OSS 20B

✅

❌

The half-sphere cut is present, but incorrectly offset.

Overall, most "full" cloud models managed to get everything right spec-wise, and included some sensible formatting. On the other hand, local models struggled with the half-sphere more often than not, with only GLM-4.5-Air managing to get it right.

It would be unfair not to mention that our setup is relatively brutal towards local models. There’s no fine-tuning, no prompt engineering beyond the base prompt, or other methods to wring more of them. Even a follow-up query to check "their work" for errors results in better output. Nevertheless, I’d argue the comparisons actually demonstrate how good current local models are.

Right, let’s proceed to the next experiment… just joking, of course we’re going to print the results!

From matrix multiplication to plastic strings

Let’s take GLM 4.5-Air’s output as an example. This is the code:

// Cube of size 10mm
cube(10);

// Extruded triangular prism with base sqrt(2)*10mm and height 10mm
s = sqrt(2) * 10;
h = (sqrt(3)/2) * s;
points = [
    [0, h * 2/3],
    [-s/2, -h/3],
    [s/2, -h/3]
];
linear_extrude(height=10) polygon(points);

// Half-sphere (hemisphere) of radius 5mm with flat face downward
difference() {
    sphere(r=5, $fn=20);
    translate([0,0,-5]) cube(10, center=true);
}

This is the initial rendering (in FreeCAD):

This is the print timelapse^[3]:

Your browser does not support the video tag.

And this is the result, printed at a "rough" 0.3mm layer height – usually reserved for functional parts – in white PLA:

Figure 3. Something something late 90s video game publisher slogan

Kinda' cool, right?

Practical prints

Design

We’ve gone through a simple scenario for initial validation. Now, onto something more practical.

3D printing, specifically FDM/FFF printing, offers some unique opportunities not present in other manufacturing methods. One of these is the ability to create intrinsically enclosed and/or enmeshed multimaterial assemblies. During the print process, we can pause at specific layers, and add/mount subcomponents providing additional capabilities.

And you know what’s annoying? When you need to assemble something – like a piece of furniture – you have your set of screws, bolts, or other small elements… so you lay them out in a more or less organized fashion. But then, then your concentration gets broken, you need to get up to retrieve something, and you (or a less or more small furry critter you house, you didn’t notice coming in) end up kicking said small things all over the place.

Which leads to the following project: a small tray with several compartments, and, critically, embedded magnets in the compartment floors – to hold the aforementioned screws, etc. down.

This time, we’re going to try both cloud and local models, but now more informally, with a limited selection, and allowing for some corrections.

Keeping things more concise, what follows is the prompt that resulted in the best initial results:

Create an OpenSCAD file that encodes a 50 mm by 50 mm tray divided into four compartments. The divider and external walls of the tray must be 5mm tall and 1.2mm wide. The base of the tray should be at least 3 mm thick, and include voids able to accommodate 4mm x 10mm x 2 mm magnets. The voids should be placed at the center of each compartment, and be no farther away than 0.6mm from the top surface of the base of the tray.

Claude Sonnet (3.7 Thinking) produced an interesting case, i.e. two solids:

Figure 4. (gist)

Gemini 2.5 Pro needed to correct a syntax error, but produced a single solid with proper magnet holes:

Figure 5. (gist)

GPT-4.1 generated a curious abberation, with some elements clearly inverted compared to what they should be:

Figure 6. (gist)

But after a couple of followups, it got there.

Figure 7. (gist)

Now onto the local models. gpt-oss-20b actually got it right on the first try – outshining the cloud’s Claude 3.5 !

Figure 8. (gist)

Qwen3-Coder-30B-A3B got off to a rocky start:

Figure 9. (gist)

It finally turned around, after almost hitting the context window limit, needing to be explicitly told what one major issue is (incorrect assumption for the location of the coordinate origin), and needed to be kept reminded not to introduce "magic numbers".

Figure 10. (gist)

Devstral got it so completely wrong that it’s difficult to say where to start:

Figure 11. (gist)

However, it only took two followups to get the correct solid. And unlike Qwen3-Coder, Devstral immediately acted on the feedback about the Z offset for the magnet pockets being incorrect.

GLM 4.5 Air is a funny case. The first result was completely wrong:

Figure 12. (gist)

and it didn’t manage to produce a sensible fix before running out of context window. It appeared to code in everything but the base as a void, and got stuck on that.

…but got it the model completely right on the very second try:

Figure 13. (gist)

Implementation

Theoretically, we’re now ready to put the models in the slicer and print something out. Practically, there is a problem, which I’m sure any reader with some experience in FFM manufacture has already noticed.

Let’s cut to the chase: none of the designs provide any allowance for:

the installed magnets having anything else than the provided dimensions,
the print itself actually not holding its desired dimensions.

The above is actually quite damning: handling tolerances is fundamental not only to 3D printing, but any manufacturing process. Barring some Angstrom-level nanofabrication^[4], nothing fits precisely. And don’t get me wrong: modern consumer 3D printers offer a crazy level of precision for their purchase price. Still, when 0 mm precision implies 1e-5 mm clearance, that’s still that many moles of plastic or metal that need to fit somewhere. Sure, most materials provide some give, but there’s a limit.

The choice of magnets as our inclusion into the gem we’re creating was quite intentional. Magnets introduce their own particular problems in 3D printing, for the virtue of being what they are. If the print bed attracts the magnets well enough (and many bed surfaces are actually magnetized), there’s the risk of the magnet ripping off the bottom layers of the object as its being removed. And if the printer’s toolhead’s nozzle is attractive to the magnet enough… congratulations, you’ve earned an achievement of a relative exotic print failure, possibly with physical printer damage.

Back in the context of LLMs, all of the above highlights one of their major flaws: their lack of perceived "understanding" of various intricacies of problem domains – usually followed by some variation of "Of course, I completely […]" and assorted simulated self-flagellation (actual course correction not guaranteed).

The flipside of the above is that chat-specialized LLMs can and do correct themselves, even with vague prompts. And, for example, Claude^[5] did exactly that, provided a prompt to correct its original output:

Review the provided code for any possible tweaks that are required, or recommended, for a 3D print of the object to succeed. Apply those tweaks to the original code, while still preserving the validity of it with respect to the original spec.

// Tray with 4 compartments and magnet voids
// Dimensions: 50mm x 50mm

// Parameters
tray_width = 50;
tray_length = 50;
wall_height = 5;
wall_thickness = 1.2;
base_thickness = 3;

magnet_width = 4;
magnet_length = 10;
magnet_height = 2;
magnet_offset_from_top = 0.6;

// Print tolerances
magnet_tolerance = 0.15; // Added clearance for easier magnet insertion
chamfer_size = 0.3; // Chamfer at void opening

// Calculate internal dimensions
inner_width = tray_width - 2 * wall_thickness;
inner_length = tray_length - 2 * wall_thickness;
compartment_width = (inner_width - wall_thickness) / 2;
compartment_length = (inner_length - wall_thickness) / 2;

module magnet_void_with_chamfer() {
    // Main void with tolerance
    cube([magnet_width + magnet_tolerance, magnet_length + magnet_tolerance, magnet_height + 0.1]);

    // Chamfer at opening
    translate([0, 0, magnet_height])
    linear_extrude(height = chamfer_size, scale = [(magnet_width + magnet_tolerance + 2*chamfer_size)/(magnet_width + magnet_tolerance), (magnet_length + magnet_tolerance + 2*chamfer_size)/(magnet_length + magnet_tolerance)])
    translate([0, 0, 0])
    square([magnet_width + magnet_tolerance, magnet_length + magnet_tolerance]);
}

module base() {
    difference() {
        // Base plate
        cube([tray_width, tray_length, base_thickness]);

        // Magnet voids in each compartment
        // Top-left
        translate([
            wall_thickness + compartment_width/2 - (magnet_width + magnet_tolerance)/2,
            wall_thickness + compartment_length/2 - (magnet_length + magnet_tolerance)/2,
            base_thickness - magnet_height - magnet_offset_from_top
        ])
        magnet_void_with_chamfer();

        // Top-right
        translate([
            wall_thickness + compartment_width + wall_thickness + compartment_width/2 - (magnet_width + magnet_tolerance)/2,
            wall_thickness + compartment_length/2 - (magnet_length + magnet_tolerance)/2,
            base_thickness - magnet_height - magnet_offset_from_top
        ])
        magnet_void_with_chamfer();

        // Bottom-left
        translate([
            wall_thickness + compartment_width/2 - (magnet_width + magnet_tolerance)/2,
            wall_thickness + compartment_length + wall_thickness + compartment_length/2 - (magnet_length + magnet_tolerance)/2,
            base_thickness - magnet_height - magnet_offset_from_top
        ])
        magnet_void_with_chamfer();

        // Bottom-right
        translate([
            wall_thickness + compartment_width + wall_thickness + compartment_width/2 - (magnet_width + magnet_tolerance)/2,
            wall_thickness + compartment_length + wall_thickness + compartment_length/2 - (magnet_length + magnet_tolerance)/2,
            base_thickness - magnet_height - magnet_offset_from_top
        ])
        magnet_void_with_chamfer();
    }
}

module walls() {
    // Outer walls
    difference() {
        cube([tray_width, tray_length, wall_height + base_thickness]);
        translate([wall_thickness, wall_thickness, base_thickness])
        cube([inner_width, inner_length, wall_height + 1]);
    }

    // Horizontal divider
    translate([wall_thickness, tray_length/2 - wall_thickness/2, base_thickness])
    cube([inner_width, wall_thickness, wall_height]);

    // Vertical divider
    translate([tray_width/2 - wall_thickness/2, wall_thickness, base_thickness])
    cube([wall_thickness, inner_length, wall_height]);
}

// Render the complete tray
base();
walls();

Summarized as:

Key improvements:

Magnet voids now 0.15mm larger for press-fit tolerance

Chamfered openings prevent filament strings and ease magnet insertion

All dimensions preserved per original spec

Print-friendly geometry maintained

That is an improvement, although one gripe remains in the tolerance size, usually dependent on the layer thickness.

A surprising outcome is the addition of chamfers: on this scale, will do sweet FA, but it shows the relevant embeddings for general 3D printing improvements are present.

Let us revert to local models. For GLM 4.5 Air, the model did attempt to make the output better:

Key Improvements for 3D Printability:

Base Thickness Increase:

Increased from 3mm to 4mm to provide more material below magnet voids

Ensures 1.4mm material below voids (vs. 0.4mm in original) for structural integrity

Still meets "at least 3mm" specification requirement

Magnet Void Positioning:

Voids now start at z=1.4mm (previously z=0.4mm) in a 4mm base

Maintains 0.6mm clearance from top surface

Preserves exact 4x10x2mm magnet dimensions

Structural Enhancements:

Eliminated thin-walled sections below magnets that would be prone to breakage

Maintained all original dimensional specifications

Kept wall thickness at 1.2mm (suitable for FDM printing)

Print Orientation Recommendations:

Print with base down (as modeled)

Supports should be added for internal divider bases if needed

Print at 0.2mm layer height for detail retention

Material Considerations:

Total height now 9mm (4mm base + 5mm walls)

Maintains compartment size (25x25mm) and wall thickness (1.2mm)

Preserves exact 50x50mm external dimensions

You’ll note the conspicuous lack of tolerance factors for the magnet voids.

Giving it more of a hint:

These are good, but miss one aspect. Here’s a hint: when inserting objects into a 3D print, are they generally the exact size specified, and are the voids in the print also of that exact size?

Using the hint, apply tweaks to the provided code, while still preserving the validity of it with respect to the original spec.

provides us with:

// Tray dimensions
tray_width = 50;    // mm
tray_length = 50;   // mm
tray_height_base = 4;  // Increased from 3mm for strength
wall_thickness = 1.2;  // mm
wall_height = 5;       // mm
compartment_size = tray_width / 2;  // 25mm each

// Magnet dimensions with tolerance for insertion
magnet_width = 4;    // mm
magnet_length = 10;  // mm
magnet_height = 2;   // mm
magnet_tolerance = 0.2;  // Added tolerance per dimension
magnet_depth_offset = 0.6;  // mm from top of base

// Effective magnet void dimensions (with tolerance)
void_width = magnet_width + 2 * magnet_tolerance;
void_length = magnet_length + 2 * magnet_tolerance;
void_height = magnet_height + magnet_tolerance;  // Only add tolerance to depth

// Calculate positions for magnet voids
magnet_x = compartment_size / 2 - magnet_length / 2;
magnet_y = compartment_size / 2 - magnet_width / 2;
magnet_z = tray_height_base - magnet_depth_offset - magnet_height;

// Generate the tray
difference() {
    // Base and walls combined
    union() {
        // Base (now 4mm thick)
        cube([tray_width, tray_length, tray_height_base], center=false);

        // External walls
        translate([0, 0, tray_height_base])
            cube([wall_thickness, tray_length, wall_height]);          // Left wall
        translate([tray_width - wall_thickness, 0, tray_height_base])
            cube([wall_thickness, tray_length, wall_height]);          // Right wall
        translate([0, 0, tray_height_base])
            cube([tray_width, wall_thickness, wall_height]);          // Bottom wall
        translate([0, tray_length - wall_thickness, tray_height_base])
            cube([tray_width, wall_thickness, wall_height]);          // Top wall

        // Internal dividers
        translate([compartment_size, 0, tray_height_base])
            cube([wall_thickness, tray_length, wall_height]);          // Vertical divider
        translate([0, compartment_size, tray_height_base])
            cube([tray_width, wall_thickness, wall_height]);          // Horizontal divider
    }

    // Magnet voids in each compartment (with tolerance)
    // Bottom-left
    translate([magnet_x - magnet_tolerance, magnet_y - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);

    // Bottom-right
    translate([magnet_x + compartment_size - magnet_tolerance, magnet_y - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);

    // Top-left
    translate([magnet_x - magnet_tolerance, magnet_y + compartment_size - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);

    // Top-right
    translate([magnet_x + compartment_size - magnet_tolerance, magnet_y + compartment_size - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);
}

The model outputs a suboptimal result, in that it quasi-hallucinates the tolerance values w.r.t. height. Nevertheless, and unlike the cloud-based model, it potentially corrects for both aspects of magnet inclusion in 3D prints.

And here’s gpt-oss' result:

//============================================================
//  50 × 50 mm Tray with 4 compartments
//  Walls: 5 mm tall, 1.2 mm wide
//  Base: 3 mm thick
//  Magnet voids (4 × 10 × 2 mm) – centred in each compartment
//  Void top is 0.6 mm from the top of the base
//============================================================

//---------------------------
//  Parameters
//---------------------------
tray_size      = 50;      // overall tray size (mm)
wall_thickness = 1.2;     // wall thickness (mm) – thin for FDM, keep spec
wall_height    = 5;       // wall height (mm)
base_thickness = 3;       // base thickness (mm)

magnet_size      = [4, 10, 2];   // [x, y, z] size of the magnet void (mm)
magnet_depth_top = 0.6;          // distance from top of base to top of void (mm)
magnet_z_bottom  = base_thickness - magnet_depth_top - magnet_size[2]; // z of void bottom (mm)

//---------------------------
//  Helper modules
//---------------------------
/// Draws a single magnet void at the given centre point
module magnet_void(centre) {
    translate([ centre[0] - magnet_size[0]/2,
                centre[1] - magnet_size[1]/2,
                magnet_z_bottom ])
        cube( magnet_size );
}

/// Base of the tray – 50 × 50 × 3 mm – with the four voids subtracted
module base_with_voids() {
    difference() {
        color("lightgrey")  // base colour (preview only)
            cube([tray_size, tray_size, base_thickness]);

        // centres of the four compartments
        centres = [
            [12.8, 12.8],   // bottom‑left
            [37.2, 12.8],   // bottom‑right
            [12.8, 37.2],   // top‑left
            [37.2, 37.2]    // top‑right
        ];

        for (c = centres)
            magnet_void(c);          // subtract the void
    }
}

/// Outer walls – 5 mm tall, 1.2 mm thick, positioned on top of the base
module outer_walls() {
    difference() {
        translate([0, 0, base_thickness])  // walls start on top of base
            color("tan")  // wall colour (preview only)
                cube([tray_size, tray_size, wall_height]);

        translate([wall_thickness, wall_thickness, base_thickness])
            cube([ tray_size - 2*wall_thickness,
                   tray_size - 2*wall_thickness,
                   wall_height ]);
    }
}

/// Two internal divider walls (vertical + horizontal)
module divider_walls() {
    // vertical divider – centred at x = 25 mm
    translate([ 25 - wall_thickness/2,
                wall_thickness,
                base_thickness ])
        color("tan")
            cube([ wall_thickness,
                   tray_size - 2*wall_thickness,
                   wall_height ]);

    // horizontal divider – centred at y = 25 mm
    translate([ wall_thickness,
                25 - wall_thickness/2,
                base_thickness ])
        color("tan")
            cube([ tray_size - 2*wall_thickness,
                   wall_thickness,
                   wall_height ]);
}

//---------------------------
//  Assemble the tray
//---------------------------
union() {
    base_with_voids();
    outer_walls();
    divider_walls();
}

While it did provide a significantly streamlined code (unprompted bells and whistles included), it also still needed an explicit reminder about the precision issue before it outputted a model with tolerances taken into account.

Right, let’s get to the printing.

Results

We’ll take the GLM 4.5 Air model for the print, with the following changes:

slightly adjusting:
- the numerical values for the tolerance,
- the offset from the top,
- and the thickness of the base;
adding an additional z clearance to the magnet void height;
and fixing the void_height calculation, which should also be based on magnet_depth_offset.

The base thickness and top offset changes, or rather, the need for them, became apparent when it turned out that the currently-used nozzle is significantly more magnetic than expected – so the magnets need to be nested deeper into the print, which will make them hopefully attracted more to the printbed than to the nozzle.

// Tray dimensions
tray_width = 50;    // mm
tray_length = 50;   // mm
tray_height_base = 6;  // Increased from 3mm for strength
wall_thickness = 1.2;  // mm
wall_height = 5;       // mm
compartment_size = tray_width / 2;  // 25mm each

// Magnet dimensions with tolerance for insertion
magnet_width = 4;    // mm
magnet_length = 10;  // mm
magnet_height = 2;   // mm
magnet_tolerance = 0.6;  // Added tolerance per dimension
magnet_z_clearance = 2.1;  // Additional clearance in height
magnet_depth_offset = 0.6;  // mm from top of base

// Effective magnet void dimensions (with tolerance)
void_width = magnet_width + 2 * magnet_tolerance;
void_length = magnet_length + 2 * magnet_tolerance;
void_height = magnet_height + magnet_tolerance + magnet_z_clearance;  // Only add tolerance to depth

// Calculate positions for magnet voids
magnet_x = compartment_size / 2 - magnet_length / 2;
magnet_y = compartment_size / 2 - magnet_width / 2;
magnet_z = tray_height_base - void_height - magnet_depth_offset;

// Generate the tray
difference() {
    // Base and walls combined
    union() {
        // Base (now 4mm thick)
        cube([tray_width, tray_length, tray_height_base], center=false);

        // External walls
        translate([0, 0, tray_height_base])
            cube([wall_thickness, tray_length, wall_height]);          // Left wall
        translate([tray_width - wall_thickness, 0, tray_height_base])
            cube([wall_thickness, tray_length, wall_height]);          // Right wall
        translate([0, 0, tray_height_base])
            cube([tray_width, wall_thickness, wall_height]);          // Bottom wall
        translate([0, tray_length - wall_thickness, tray_height_base])
            cube([tray_width, wall_thickness, wall_height]);          // Top wall

        // Internal dividers
        translate([compartment_size, 0, tray_height_base])
            cube([wall_thickness, tray_length, wall_height]);          // Vertical divider
        translate([0, compartment_size, tray_height_base])
            cube([tray_width, wall_thickness, wall_height]);          // Horizontal divider
    }

    // Magnet voids in each compartment (with tolerance)
    // Bottom-left
    translate([magnet_x - magnet_tolerance, magnet_y - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);

    // Bottom-right
    translate([magnet_x + compartment_size - magnet_tolerance, magnet_y - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);

    // Top-left
    translate([magnet_x - magnet_tolerance, magnet_y + compartment_size - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);

    // Top-right
    translate([magnet_x + compartment_size - magnet_tolerance, magnet_y + compartment_size - magnet_tolerance, magnet_z])
        cube([void_length, void_width, void_height]);
}

We export this code into and STL file, and examine it in the slicer:

Figure 14. Typical slicer program preview. The slider on the right allows to move through the layers, and the one on the bottom scrolls the printer nozzle’s simulated position within a layer.

Here’s how the layer transitions look like:

Transition of the print throughout the layers. Note the legend on the right.

Your browser does not support the video tag.

The magnet voids appear to be placed between layers 2 and 18 (inclusive), which, for a 0.3 mm layer height, means that there is indeed the correct amount of space. Note that is a "Pause" marker present at layer 18: this was done manually, to be able to insert the magnets without an oven-hot nozzle assembly, backed by surprisingly strong motors, hitting one’s hand at speed.

Now, what’s left is to print:

Your browser does not support the video tag.

And does it work?

Your browser does not support the video tag.

It does!

Code Review

As a final opportunity for insights, let’s go back to the various pieces of generated code, to see:

what can we learn about the OpenSCAD language itself,
what patterns are present, and how frequently they appear.

This is an area where the LLMs' usual propensity for overcommenting actually works in our favor. In general, we can see that OpenSCAD offers a set of primitives used together to formulate the final assembly. There’s also variables, and, for code organization, modules (functions).

Interestingly, for loops also exist (example, cf. tutorial, also list comprehensions), but are usually avoided, unless directly or indirectly prompted for. The net effect is the code becoming more verbose, and not actually always easier to follow. Furthermore, "unrolling" loops is the preferred style in OpenSCAD’s tutorial – which likely formed a significant part of the relevant training data available for OpenSCAD code.

The models also consistently make similar mistakes:

introducing "magic numbers" instead of fixing every value into a variable,
making "origin-related" offsetting errors,
"forgetting" or juxtaposing elements in the calculations (such as our GLM-4.5-Air code),
"fluffing up" the code with unnecessary style elements (like colors),
etc.

Luckily, at least at the level of complexity we’re dealing with, these mistakes are easy to spot by anyone with a half-decent eye for correctness in programs.

Outro

Hopefully, this exploration has provided not only a highlight of the fact that code-focused LLMs can generate infographics or functional 3D models, but also some measure of inspiration to check out what is also possible, without resorting to more specialized LLMs.

It is also quite interesting that even vastly simpler, local models can mostly keep up with their cloud-based counterparts, at least for this type of task.

On the flip side, the fact that everything covered here is ultimately expressed in some form of code provides a distinct advantage over typical "genAI" fare. Code can be inspected and, most importantly, easily modified, unlike, say, a typical raster image. Even the verification element is extra-useful here, as such observations can be converted into follow-up prompts to correct the output (at the risk of being you’re-absolutely-righted again).

And that’s about it for this article. I do encourage, in case anyone finds similar use cases, to share them in the comments.

1. Graphics design is my passion 🐸.

2. Neither is it completely representative of practical usage, e.g. the tokens-per-second performance varies considerably across models.

3. The solids were, of course, spaced out during the export process.

4. Yes, I do realize that’s an oxymoron, thank you very much :).

5. which I would love to say was 3.7 Thinking, but, because Copilot arbitrarily cut access to that in the meantime, was actually 4.5

Extracting Machine Learning Training Data from a Video Game – Using "Ready-Mades"

Sat, 12 Oct 2024 00:00:00 +0200

Table of Contents

Starting words
Modern Detectiving with Open-Vocabulary Models
- Intro
- Implementation
Local Large Models to the rescue?
Do we even need the designator? Looking into auto-segmentation via SAM 2
"Mainstream" LLMs
- Claude
- ChatGPT
Summary

Illustration adapted from photo by Sven Brandsma on Unsplash

Starting words

Hello again! We are continuing our series on extracting Machine Learning training data from a video game. In the previous blog entry, we’ve run some exploratory data analysis on what we aim to extract for the "final" model’s training. From now on, we’ll be focusing on the actual extraction process of said data.

The theme of this particular post is "ready-made". In other words, we’re going to look at some relatively current methods to solve our problem – or side step it; ones that have a characteristic of doing most of the work for us. We’ll start with a modern detection model, and then proceed with a local LLM (or rather, VLM), explore alternative sources of auto-derived detection training data, and compare all that with hosted LLMs from leading vendors.

The current entry is likely to be the least "technically complex" in the entire series, meaning it will be the easiest to replicate for an arbitrary person, and therefore adapt to other use cases. Something to keep in mind, especially if you, Dear Reader, stumbled upon this post randomly.

For a refresher on what we want to achieve, feel free to consult the starter blog entry, here.

Modern Detectiving with Open-Vocabulary Models

Intro

"Modern" is, of course, an ill-defined term in the current breakneck-paced bazaar of ML solutions. However, we can provide a sensible generalization of what that could be at the moment of writing of this entry, at least for our specific problem scope.

Speaking of specific, one relatively recent – i.e., no more than 2-year-old – trend for detection models is to enable operating on an "open vocabulary", as opposed to a rigid set of classes. So, instead of telling the model to detect classes of objects like "car", "bicycle", and similar, the user can be more creative, and supply prompts such as "yellow vehicle", or "car with two windows".

As already mentioned, open-vocabulary detection models have been developed for about 2 years now. A relatively fresh example, and one we’ll use in the current section, is YOLO-World, released early 2024. True to its name, it is based on YOLO for detection, augmented by the ability to fuse text and image embeddings (representations of both in a numeric vector space). A detailed explanation is beyond the scope of this blog – for those interested, the original paper is available on arXiv.

We’ll now see if we can wrangle YOLO-World to detect our designators. To juggle our memory from the earlier entries, the designator is depicted in the center of the screenshot below, surrounding the targeted mech:

Figure 1. The "base image" we’ll use for detection and similar tasks, offering a relatively uncluttered scene with both the designator and a mech clearly in view.

Implementation

The nice thing about testing out ML models in recent years is that, for the vast majority of use cases, everything is so, so convenient^[1]. Not only are there repositories that standardize model usage and deployment – most prominent Hugging Face, but they often come "free" with libraries further expediting the use of models present within their ecosystem. On top of that, utility libraries exist that aggregate those "commonized" APIs for model usage, evaluation, and similar tasks into a meta-API, so that an arbitrary model authors' choice of standards is not as much of a pain point as it was, say, 5 years ago (yes, yes, feel free to add your favorite joke on standards proliferation here).

A library/framework that possesses the quality praised in the preceding paragraph, and one that we’ll be using here, is Supervision.

So let’s see how that goes; by "that" we mean using YOLO-World to find the target designator in the image. We’ll essentially be following along one of Supervision’s tutorials, with some small modifications:

import cv2 as cv
import supervision as sv
from supervision import Position

from inference.models.yolo_world.yolo_world import YOLOWorld

detection_model = YOLOWorld(model_id="yolo_world/l")

# several different class prompts, in the hope that at least
# one will match for the designator
classes = ["red rectangle", "red outline", "red box", "red designator"]

detection_model.set_classes(classes)

# setting up the annotators
BOUNDING_BOX_ANNOTATOR = sv.BoundingBoxAnnotator(thickness=2)
LABEL_ANNOTATOR = sv.LabelAnnotator(text_thickness=1, text_scale=0.5, text_color=sv.Color.WHITE, text_position=Position.BOTTOM_LEFT)

# loading the image shown above
frame = cv.imread("example_designator.jpg")

# we're using a very low confidence threshold, as we're
# interested in seeing "what sticks"
results = detection_model.infer(frame, confidence=1e-3)

# however, we are also still applying NMS, as potentially, in low
# confidence scenarios, we run the risk of being inundated with multiple,
# redundant, tiny detections
detections = sv.Detections.from_inference(results).with_nms(threshold=1e-4)

print(detections)

# will print out something like:
# Detections(xyxy=array([[     709.02,      829.93,      810.31,      1055.4],
#       [     810.56,      343.53,      879.66,      390.26],
#       [     799.74,       807.5,      1123.7,      1063.1],
#       [     809.68,      343.99,      879.36,      390.05]]),
#       mask=None,
#       confidence=array([  0.0019695,   0.0014907,   0.0012708,   0.0012423]),
#       class_id=array([2, 2, 2, 0]),
#       tracker_id=None,
#       data={'class_name': array(['red box', 'red box', 'red box', 'red rectangle'], dtype='<U13')})


# this is, again, pretty much copied from the linked tutorial
# BTW, it is a bit unusual there's no sensible
# way to compose Supervision annotators

annotated_image = frame.copy()

labels = [
    f"{classes[class_id]} {confidence:0.5f}"
    for class_id, confidence
    in zip(detections.class_id, detections.confidence)
]


annotated_image = BOUNDING_BOX_ANNOTATOR.annotate(annotated_image, detections)
annotated_image = LABEL_ANNOTATOR.annotate(annotated_image, detections, labels=labels)
sv.plot_image(annotated_image, (30, 30))

The final line results in the following image:

Figure 2. Results of initial YOLO-World detection attempts.

Well, that’s not looking great. The model, although powerful, evidently has trouble distinguishing various elements in our screenshot. The likely underlying reason is simple — the model was trained on datasets of "actual" images, i.e., photographs, and not on game screenshots^[2].

Instead of looking at multiple (pre-)training datasets, we’ll check out the one used for evaluation, specifically LVIS. Let’s take a small look at that dataset to see what kind of text input data it provides. The spec is here – we’re interested in the "Data Format" → "Categories" section in particular. Let’s also load up the training set and see what some example data looks like. Fire up this code and read the spec in the time it takes to load:

# download the file from https://dl.fbaipublicfiles.com/LVIS/lvis_v1_train.json.zip
# and unzip it to the current directory

import pandas as pd
import json

with open("lvis_v1_train.json") as fp:
    lvis_training_instances = json.load(fp)

print(list(lvis_training_instances.keys()))
# prints out ['info', 'annotations', 'images', 'licenses', 'categories']

categories_df = pd.DataFrame(lvis_training_instances["categories"])

categories_df

Figure 3. Example of the LVIS dataset categories.

Feel free to explore both the names and the synonyms of the categories – the latter obtainable via a snippet like the following one^[3]:

category_names_synonyms = categories_df['synonyms'].explode().unique()

It will become quickly apparent that the problem lies in, well, the problem domain of the model – actual photographs of real-world objects. MechWarrior: Online is pretty far from being photorealistic (both due to stylistic choices and age), so the scenes in the screenshot can’t always be meaningfully interpreted by the model in the context of its training data, even down to basic visual features. Demonstrating the latter problem is the following query, attempting to capture the red landing lights visible throughout the screenshot:

classes = ["red light", "red lightbulb", "landing light"]

None of the results remotely capture what we intended. The "big" detection is likely due to an association with a… related concept, which we can verify by changing the classes appropriately:

classes = ["airplane", "shuttle", "vehicle"]

Yep, the model actually does manage to "recognize" the DropShip visible in the screenshot as an airplane^[4]. Note the considerably higher confidence – 0.8 is something on the level you would expect from an "actual" detection, as opposed to the unusually low confidence we used for the investigation.

Evidently, YOLO-World is actually capable of detecting artificially generated visuals, but not what we require^[5]. In fact, we’ll see similar trends with other SotA (and non-SotA) models: virtually always, training datasets include COCO, Object365, and so on. The "mainstream" models are, in general, not prepared to operate on rendered images, at least not specifically so.

So what can we do with this? One way to go is to adapt the model to our needs.

…and this is what we would have proceeded with, had we not already declared that the blog entry will not be overtly technical. Instead, we’ll perhaps revisit the adaptation task in another entry, but, for now, we’ll just chalk this up as a lesson that even "generalist" models aren’t general enough for each and every use case – context still matters.

Local Large Models to the rescue?

Fortunately, we still have several "levels" to go up on, the first one being trying out a more powerful, but still locally runnable^[6], model.

Moondream, the one we’ll use, is described as a "Vision Language Model". Its inputs are both an image, and a text prompt. The latter can be anything from a simple "describe this image" request, to "what is wrong with this specific object in the picture" quasi-anomaly-detection. In between, we have an option to request detection bounding boxes for objects described in a freeform manner, and this is what we’ll use.

As already alluded to, the model is relatively small for an LLM/VLM, and can easily be run on even a laptop-grade GPU, while still having decent time performance. The model is available on Hugging Face, which makes getting it work a breeze. Curiously, there’s little more information about it; publication-wise, the only references that can be found to the model come from comparison papers, such as this one^[7].

Moondream: initial approach and examination

Regardless, let’s get right to working on it, following the example code and our test screenshot, with some small modifications:

from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image

# if GPU
DEVICE = 'cuda'
# uncomment if GPU insufficient
# DEVICE = None

model_id = "vikhyatk/moondream2"
revision = "2024-08-26"
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, revision=revision, device_map=DEVICE
)
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)

base_image = Image.open('blog/img/intro_interface_demo_raw/frame_5.jpg')


def infer(image, question):
    enc_image = model.encode_image(image)
    return model.answer_question(enc_image, question, tokenizer)


print(infer(base_image, "Describe this image."))

This will give us the following:

The image shows a screenshot from a video game, featuring a player’s view of a futuristic environment with a large vehicle, mountains, and a control panel with various game elements.

Impressive, isn’t it? Five years ago, models of this apparent analytical complexity, running on local hardware, would still be considered science-fiction.

To cut the optimistic tone somewhat, note that we have little amounting to specifics – just a general description of the scene. The output does mention the hills in the background, and a vehicle, but that’s about it.

Wait a minute 'though: maybe the "vehicle" is what we need? To figure it out, we need to write a simple function that annotates the image with bounding boxes resultant from the model’s output. And yes, the model is capable of outputting bounding box coordinates. Here’s the code:

import ast
import numpy as np
import supervision as sv

def bbox_to_annotation(image, prompt):
    bbox_str = infer(image, prompt)

    # this is unsafe in general, especially when parsing
    # open-ended model outputs! Used here for demo purposes only.
    bbox = ast.literal_eval(bbox_str)

    # retrieve the xyxy coordinates
    x1, y1, x2, y2 = bbox

    # the coordinates are relative to the image size,
    # convert them to pixel values
    x1, x2 = (np.array([x1, x2]) * image.size[0]).astype(int)
    y1, y2 = (np.array([y1, y2]) * image.size[1]).astype(int)

    # create a Supervision Detections object with just our single bounding box
    detections = sv.Detections(np.array([[x1, y1, x2, y2]]), class_id=np.array([0]))

    # set up the annotator, which is a Supervision API for convenient
    # drawing, swapping, and composing various annotation types
    annotator = sv.BoxAnnotator()

    # need to copy the image – annotator works in-place by default!
    return annotator.annotate(image.copy(), detections)

And here’s the invocation with the result:

bbox_to_annotation(base_image, "Bounding box of the vehicle.")

Figure 4. Moondream’s detection result for the base image and the "vehicle" prompt.

The "vehicle" turned out to be the large DropShip craft sitting on the runway, so that' a miss.

Let’s continue by exploring what the model "sees" in the vicinity of the target designator, as this might give a better idea of what vocabulary we might have to use for a successful prompt. First, directly, by cropping the image to just the designator.

Figure 5. Image being analyzed below.

designator_location_base_image = [860, 418, 955, 512]

base_image_designator_crop = sv.crop_image(base_image, xyxy=designator_location_base_image)
infer(base_image_designator_crop, "Describe the image")

The image features a robot with a dark silhouette, standing in a red-framed area. The robot appears to be in a defensive stance, possibly ready to attack.

Apart from the actually internally consistent, but still amusing expression of "defensive stance, […] ready to attack", two observations are of note here:

the model does seem to actually recognize mechs as "robots", which is impressive in and of itself;
crucially for us, it also notices a "red-frame area", i.e., our designator.

We’ll eventually return to the former, proceeding now with the latter. We know that the model is able to associate the relevant UI element with text embedding in its encoding space that results in the "red-frame". We should now determine how sensitive the model is to this "stimulus" when given a broader context. To do that in a low-tech fashion, we’ll repeatedly run the inference on an image consisting of the designator, plus a variable bit of margin. The variability will span from a relatively small dilation factor of several pixels, up to most of the image in question.

The code to perform the task is pretty simple:

def random_crop(image, base_bb, offsets_range_max: int):
    # convert to np.array to allow for vectorized operations
    bb_xyxy = np.array(base_bb)

    # define the "directions" in which the offsets are applied
    # first corner should go only "up" and "left",
    # second corner should go only "down" and "right"
    offset_direction = [-1, -1, 1, 1]

    # generate the random offsets for all BB coordinates
    offsets = np.random.randint(0, offsets_range_max, size=len(base_bb))

    # perform the "widening" calculation on the original BB
    crop_box = bb_xyxy + offset_direction * offsets

    # ensure the resultant crop BB is within the image's bounds
    repeat_max_by = len(crop_box) // len(image.size)
    clipped_crop_box = np.clip(crop_box, 0, list(image.size) * repeat_max_by)

    return sv.crop_image(image, clipped_crop_box)

Here’s an example of invocation and result:

random_crop(base_image, designator_location_base_image, 100)

Using the function we just defined, we can now generate the descriptions in the manner we discussed with the following code:

import pandas as pd
from tqdm import tqdm


def extract_descriptions(
    source_image,
    designator_xyxy,
    min_offset=1,
    max_offset=1000,
    interval=10,
    num_iter=10,
):
    description_data = []
    for offset in tqdm(range(min_offset, max_offset, interval)):
        for iter in range(num_iter):
            cropped_image = random_crop(source_image, designator_xyxy, offset)
            desc = infer(cropped_image, "Describe the image.")
            description_data.append(
                {"max_offset": offset, "size": cropped_image.size, "description": desc}
            )

    return pd.DataFrame(description_data)

description_df =  extract_descriptions(base_image, designator_location_base_image)

Here is the CSV file of the results obtained after an example run.

Figure 6. Smallest…

Figure 7. …and largest maximum offsets.

Looking over the data, it is clear that the model recognizes the designator as some element, only for its importance to fall of the description inclusion threshold it in the context of the larger picture.

Regardless, it seems that the model recognizes the frame as a "red" something, as evidenced by this diagram:

import seaborn as sns

_SEARCH_FOR = "red"

# filter the rows that contain the search term
max_offset_with_red = description_df[
    description_df["description"].str.contains(_SEARCH_FOR)
]["max_offset"].rename(f"max_offset_{_SEARCH_FOR}")

# plot against all max offsets
sns.histplot(
    [max_offset_with_red, description_df["max_offset"]],
    palette=["red", "#bbbbbb"],
    bins=10,
)

Figure 8. Histogram showing a distribution of the max_offset values where the word "red" is contained in the result. Note how the frequency of the word’s presence decreases with max_offset 's value – in other words, with the size of the visible area around the designator.

Let’s run some aggregation now, so that we may see some trends in the output. We’ll proceed with that by processing the descriptions into a quasi-canonical form, and find the word "red" in the description, along with two neighboring words on each side, and then group the results. For the NLP processing, we’ll use spacy, the documentation of which contains a very nice usage primer, also explaining some basic NLP concepts.

from functools import partial
from typing import Optional
import spacy

_RED_VICINITY = 2
_SEARCH_FOR = "red"
_FIELD_DESCRIPTION = "description"

nlp = spacy.load("en_core_web_sm")


def word_neighborhood(
    source: str, lemmatized_word: str, neighborhood_size: int
) -> Optional[str]:
    """Takes a single description, runs basic NLP to obtain lemmatized sentences, and extracts
    `neighborhood_size` words around the `lemmatized_word`, including the latter."""

    # run the basic NLP pipeline
    doc = nlp(source)

    try:
        # we assume there's only one sentence that has the word
        # not a fan of exception-driven logic, but it's cleaner in this case
        word_sentence = [
            s for s in doc.sents if any(t.lemma_ == lemmatized_word for t in s)
        ][0]
    except IndexError:
        return None

    # get the lemmatized version of the sentence, without
    # stopwords and punctuation
    processed_sentence = [
        t.lemma_ for t in word_sentence if not t.is_stop and t.is_alpha
    ]

    word_pos = processed_sentence.index(lemmatized_word)

    # an alternative would be to use the various Matcher facilities in spaCy,
    # but the chosen approach is a bit less cumbersome, doesn't require as much
    # knowledge of spacy to read, and we don't care for efficiency in this case
    return " ".join(
        processed_sentence[
            max(0, word_pos - neighborhood_size) : word_pos + neighborhood_size + 1
        ]
    )


def process_description(description_df: pd.DataFrame) -> pd.DataFrame:
    # apply the word neighborhood function to the description column
    with_red_vicinity = description_df.copy()
    with_red_vicinity[_FIELD_DESCRIPTION] = with_red_vicinity[_FIELD_DESCRIPTION].apply(
        partial(
            word_neighborhood,
            lemmatized_word=_SEARCH_FOR,
            neighborhood_size=_RED_VICINITY,
        )
    )

    # group by description and aggregate the other fields
    return (
        with_red_vicinity.groupby(_FIELD_DESCRIPTION)
        .agg(
            {
                "max_offset": np.median,
                "size": list,
                # Add other fields as needed
            }
        )
        .reset_index()
    )


with_red_vicinity_unique = process_description(description_df)

with_red_vicinity_unique

Figure 9. Top 20 grouped smallest offsets…

Figure 10. …and top 20 largest ones.

Unfortunately, we confirm the trend we were seeing in the "raw" data – the designator becomes less "distinct" in larger inputs.

In other words, we can try to use the "small detail" text for our BB prompt, but we will not get expected results:

bbox_to_annotation(base_image, "Bounding box of the red square frame border.")

Figure 11. The BB is at the very bottom-left, not where we expect it to be.

"Prompt engineering" of input images

An alternative way for us to make it easier for the model to "focus" on what we want is to limit the information available in the scene. To do that, we’ll use the insights from the previous entry to simply color-threshold the input image.

Specifically, we’ll mask everything below the 90th percentile of the color threshold for the designators we’ve analyzed, which is color value 177 for the red channel. One way to do this would be to use OpenCV^[8]. Here are the relevant code snippets and inference results:

import cv2 as cv

def mask_red_channel_opencv(image, threshold=177):

    # using the inRange function, which is slightly more readable than
    # creating a mask just based on numpy operations, i.e., something like this:
    # mask = (image[:, :, 2] >= threshold).astype(np.uint8) * 255

    mask = cv.inRange(image, (0, 0, threshold), (255,) * image.shape[-1])

    return image & np.expand_dims(mask, axis=-1)

Figure 12. Color thresholded image.

infer(masked_image, "Describe the image.")

The image is a screenshot from a video game featuring a dark background with various elements and graphics. It appears to be a screenshot from a space shooter game, possibly from the "URBANMECHE K-9" series.

bbox_to_annotation(
    masked_image,
    "Bounding box of the small red frame area.",
)

Figure 13. Designator-detection attempt after color thresholding.

That’s more like it, but still, not perfect. Moreover, the base image is pretty much the "ideal" screenshot. For a more busy scene…

masked_image_busy = sv.cv2_to_pillow(
        mask_red_channel_opencv(sv.pillow_to_cv2(busy_image))
    )

Figure 14. Color thresholded image - "busy" scene.

infer(masked_image_busy, "Describe the image.")

The image is a screenshot from a video game featuring a space theme, with various elements of a spaceship cockpit and a cityscape in the background.

bbox_to_annotation(
    masked_image_busy,
    "Bounding box of the small red frame area.",
)

Figure 15. In case you can’t see the location of the bounding box – it’s almost the entirety of the image.

No dice here, either.

Conclusions for Moondream

Like with YOLO-World, it looks like there’s simply a mismatch between what the model was trained to detect, and what we want from it. Perhaps there is a specific prompt that does let the model identify the target designator flawlessly. I’m sure that if there is, I will be made aware of it in 5 minutes of posting this blog entry. However, that prompt certainly can’t be described as easily discoverable.

Again, similarly to YOLO-World, Moondream can certainly be fine-tuned to output what we want, but that is out of scope for this entry, as defined in the introduction.

It would be remiss to leave our consideration of Moondream, and similar distilled LVMs, at this. One thing must be stressed – the models are considerably more powerful than what we’ve shown so far.

Case in point:

bbox_to_annotation(
    busy_image,
    "Bounding box of the mech on the left.",
)

bbox_to_annotation(
    busy_image,
    "Bounding box of the mech on the right.",
)

The two prompt results show:

Not only does the model readily recognize "robot-like" shapes;
it also has an embedding mapping rich enough that it can directly map the phrase "mech" onto these shapes;
the detected area actually does roughly correspond to the mechs in the scene.

All this is to say: Moondream, and models like it, are certainly powerful enough to potentially serve our main purpose, i.e., detecting the mechs on the screen, with only some tweaking. We’ll revisit this opportunity space in future entries.

Do we even need the designator? Looking into auto-segmentation via SAM 2

Throughout the current body of the blog series, we’ve operated on an assumption that extracting the designator images is our best bet of building a training set for the "final" mech detection model. Is that necessarily the case?

Those knowing a certain "law" already know the answer. Indeed, one of the changes in recent years has been the appearance of modern generalized image segmentation models^[9]. The output thereof can be then used as training data for the target detection models.

We’ll try out a relatively recent one, Segment Anything 2 from Meta (née Facebook), available alternatively on Hugging Face. There’s also a rather impressive landing page, with various demos, including interactive ones.

Perusing the docs, notebooks, etc., we should notice quite quickly what the major trait is of the model, setting it apart from the "classical" segmentation models. Namely, it can provide complete, or near-complete segmentation of an image without any other input, as evidenced by the screenshot below…

Figure 16. Our base image, auto-masked by SAM 2. Generated by following example code from this notebook, specifically the "Automatic mask generation options" section.

For comparison: the same image, segmented using SLIC, a technique introduced in 2012. Note that the segmentation provides our desired result as well in this case; in other words, the ability of SAM 2 to just fully segment the image to our spec is not the thing we’re looking for.

Its real power lies in the ability to segment images based on a point prompt. Well, that and being able to process videos based on that original prompt (and being resilient against occlusions, and other things…).

Let’s see a little demonstration. The code here is heavily derived from Roboflow’s blog entry, authored by Piotr Skalski, on SAM 2, the primary adaptation being generalizing most of the logic into a single-function API^[10]:

# again, this is mostly code from
# https://blog.roboflow.com/sam-2-video-segmentation/
# reorganized and slightly amended

import cv2 as cv
import numpy as np
import pandas as pd
import supervision as sv

from pathlib import Path

def get_masks_as_detections(object_ids, mask_logits) -> sv.Detections:
    """Converts SAM 2 output to Supervision Detections."""
    masks = (mask_logits > 0.0).cpu().numpy()
    N, X, H, W = masks.shape
    masks = masks.reshape(N * X, H, W)

    return sv.Detections(
        xyxy=sv.mask_to_xyxy(masks=masks), mask=masks, tracker_id=np.array(object_ids)
    )


def add_points_to_predictor(predictor, inference_state, point_data: pd.DataFrame):
    """Add points to SAM 2's inference state.

    This function assumes it receives a DF with the columns:
    ["point_xy", "object_id", "label", "frame_id"]

    where "label" is 1 or 0 for a pos/neg example.
    """

    # aggregate by frame and object, to conform to batching modus
    # of predictor.add_new_points
    aggregated = point_data.groupby(["frame_id", "object_id"]).agg(list)

    for (frame_id, object_id), row in aggregated.iterrows():

        points = np.array(row["point_xy"], dtype=np.float32)
        labels = np.array(row["label"])

        predictor.add_new_points(
            inference_state=inference_state,
            frame_idx=frame_id,
            obj_id=object_id,
            points=points,
            labels=labels,
        )

def ensure_frames_generated(source_video: Path, frame_dir_root: Path) -> Path:
    """Checks if the frame images, necessary for SAM 2's state init and processing, are present.

    If not, generates and saves them to the corresponding `frame_dir_root` subdirectory.
    """

    video_name = source_video.stem

    frame_dir = frame_dir_root / video_name

    # a simple check - if the video for the directory exists,
    # we assume the frames are already generated
    if not frame_dir.exists():
        sink = sv.ImageSink(target_dir_path=frame_dir, image_name_pattern="{:04d}.jpeg")

        with sink:
            for frame in sv.get_video_frames_generator(str(source_video)):
                sink.save_image(frame)

    return frame_dir


def segment_video(
    predictor,
    source_video: Path,
    point_data: pd.DataFrame,
    target_video: Path,
    label_annotator: sv.LabelAnnotator,
    mask_annotator: sv.MaskAnnotator,
    frame_dir_root: Path = Path("./video_frames"),
):
    frame_dir = ensure_frames_generated(source_video, frame_dir_root)

    # init model's state on the video's frames
    inference_state = predictor.init_state(video_path=str(frame_dir))

    # add segment guidance points to the model
    add_points_to_predictor(predictor, inference_state, point_data)

    video_info = sv.VideoInfo.from_video_path(str(source_video))

    frames_paths = sorted(
        sv.list_files_with_extensions(directory=frame_dir, extensions=["jpeg"])
    )

    with sv.VideoSink(str(target_video), video_info=video_info) as sink:
        for frame_i, object_ids, mask_logits in predictor.propagate_in_video(
            inference_state
        ):
            frame = cv.imread(str(frames_paths[frame_i]))

            detections = get_masks_as_detections(object_ids, mask_logits)

            frame_annotated = label_annotator.annotate(frame, detections)
            frame_annotated = mask_annotator.annotate(frame_annotated, detections)
            sink.write_frame(frame_annotated)

which can be invoked like this:

segment_video(
    predictor,
    source_video=Path(SOURCE_VIDEO),
    point_data=point_data,
    target_video=Path(TARGET_VIDEO_PATH),
    label_annotator=sv.LabelAnnotator(
        color=sv.ColorPalette.DEFAULT,
        color_lookup=sv.ColorLookup.TRACK,
        text_color=sv.Color.BLACK,
        text_scale=0.5,
        text_padding=1,
        text_position=sv.Position.CENTER_OF_MASS,
    ),
    mask_annotator=sv.MaskAnnotator(
        color=sv.ColorPalette.DEFAULT, color_lookup=sv.ColorLookup.TRACK
    ),
)

The predictor value is the result of a sam2.build_sam.build_sam2_video_predictor call, the instructions for setting up which are available in the relevant documentation. In all examples here, we are using the "large" model variant.

Since we’re switching from images to video, we need to actually choose a video snippet. We’ll go with this one:

As you can see, the video can potentially be challenging to segment – the color palette is muted and homogenous, the objects of interest are either small, occluded, or blend with the background well. Nevertheless, let’s try marking a single point – specifically on the very fist visible mech up in front – and see how SAM 2 performs:

That’s actually impressive. The model does manage to keep up with both the objects and the camera’s movement, only "picking up" some extraneous elements, but never extending the mask to an unacceptable level. The only bigger hangup is it losing the object and switching tracking to another one after the zoom level switch – again, nothing concerning, as there’s no indication SAM 2 should be resilient against that.

OK, so we’ve covered a single object, but, in this snippet, a grand total of 5^[11] are available to segment. We extend the point data to the following form:

import pandas as pd

# as a reminder, "label" denotes whether
# the example is "positive" or "negative"
#
# we identify our "segmentation tracks"
# by the "object_id" field

point_data = pd.DataFrame(
    [
        [[1122, 347], 1, 1, 52],
        [[1167, 348], 2, 1, 52],
        [[1030, 350], 1, 1, 76],
        [[880, 338], 2, 1, 113],
        [[900, 355], 2, 0, 113],
        [[1675, 435], 3, 1, 130],
        [[1508, 446], 4, 1, 131],
        [[1553, 444], 4, 0, 131],
        [[1867, 427], 5, 1, 145],
        [[1202, 391], 4, 1, 363],
        [[1258, 435], 3, 1, 363],
        [[755, 324], 2, 1, 393],
        [[1145, 399], 4, 1, 436],
        [[1155, 331], 4, 1, 436],
        [[1062, 383], 4, 1, 615],
        [[1011, 391], 3, 1, 636],
    ],
    columns=["point_xy", "object_id", "label", "frame_id"],
)

For reference and visualization, here is the representation of the example points on the respective frames, as well as the initial segmentation provided by SAM 2’s model state.

Figure 17. The debug frames. Note the subtitles. Example points are denoted as circles – filled for "positive", hollow for "negative". Masks generated from the model’s inference state while adding the points are also provided. Click here for a larger version.

We can see that the static image masking does look promising. The biggest problem is that SAM 2 is a bit "greedy", and often incorporates parts of the background – see especially the first frame presented – hence the need for several negative examples. We also had to provide additional example points for objects that "started out" adjacent, or even overlapping. Again, still a good show so far, given the difficulty of the input video.

From the examples provided, we get the following segmentation results:

Comparing to the single-point example, the most notable phenomenon is a markedly reduced tracking ability of the "original" object (denoted as 0). This is likely due to attempting simultaneous segmentation on another mech (denoted as 1) that’s passing right behind 0. Overall, the model is unable to persistently track virtually any of the objects without further point examples, and even then, it eventually loses persistence. Interestingly, it does manage persist the tracking of one object over the zoom sequence (4), which means SAM 2’s anti-occlusion facilities are quite powerful.

Nevertheless, we must remind ourselves that we are not investigating SAM 2 for tracking capabilities, but to provide a source of training data for detection. And, for this purpose, it seems like a promising direction. The model produces masks that are almost always more precise than the equivalent of the target designator’s shape (something we haven’t talked about yet – even when we do get the designators, a cleanup step will still be necessary before we proceed with actual training, or fine-tuning, of a detection model).

Additionally, we’ve imposed quite a challenge on SAM 2. There’s plenty of "real world" use cases where the model would be likely to perform with much greater fidelity, and so provide even better annotation data.

The only gripe is resource consumption – even the "tiny" version of SAM 2 uses too much VRAM for most consumer GPUs. The hope lies in the larger community creating derivatives that require less memory, while having similar performance.

"Mainstream" LLMs

While things were a bit different even a year ago (due to lacking image-assisted prompt possibilities), it would now be remiss to not include a comparison of processing through the "mainstream" LLMs. We’re going to take a look at two families of them: Anthropic’s Claude and OpenAI’s GPT-4o.

The plan is as follows – first, we’re going to supply the demo image with the following prompt:

The image is a screenshot of a video game. On the screenshot, there is a target designator, shaped like a square with middle portions of its sides missing, all colored red. Provide the bounding box pixel coordinates of this designator.

We’re going to repeat the same text prompt with the "busy" image. Finally, we’re going to use one of the frame images from the test video, i.e., this one:

Figure 18. The reference frame we will be using, or "the third image".

and the prompt:

The image is a screenshot from a video game in the BattleTech/MechWarrior franchise. Provide bounding box locations of all the mechs in the image.

We’ll adapt this preliminary plan in case a potential for additional insights manifests itself.

Claude

Spoiler: this won’t go well. Starting with demo image and the specified prompt:

I apologize, but I’m not able to provide exact pixel coordinates from the image. However, I can describe the location and appearance of the red target designator you’re referring to:

The red target designator is located roughly in the center of the image, slightly below the midpoint vertically. It’s surrounding what appears to be a mech or large robotic figure standing on a road or runway.

The designator consists of a square shape with the middle portions of each side missing, creating a crosshair-like effect. Inside this designator is a distance readout showing "478m".

If you need precise pixel coordinates, you would need to use an image editing tool or specialized software to analyze the image. I can only provide descriptive information based on what I see in the image.

(emphasis added)

So, unfortunately, it looks like Claude immediately turns out to be a dud for our purposes. Now, it’s not all bad – it does recognize both the designator and the mech in the image. We note also that Claude considers the distance readout as being inside the designator; we’ll get back to this observation in the ChatGPT section.

Just not to end Claude’s exploration too abruptly, let’s see what it has to say for some other images and prompts. Here’s what it outputs when provided the "busy" image, with an adapted prompt:

Unfortunately, Claude doesn’t seem to recognize the mech that’s actually being targeted – there was a bit of hope the designator itself would provide sufficient context. At least it does identify the two other relevant objects in view. However, it describes the one on the left erroneously as hostile – quite obviously, due to the "friendly" marker above visually separated from the mech by one of the UI elements.

Interestingly, Claude does appear to have some information about the franchise, as it correctly references the existence of the assault weight class^[12]. Not surprising, 'though – BattleTech is enjoying its 40-year anniversary right about now.

Finally, let’s see if Claude can at least recognize mechs in a more challenging screenshot:

Disappointing as well: the correct answer is 5, not counting the player.

Before we conclude, we must note that the prompts were answered by the free version of Sonnet. While this may seem unfair, Claude all but disqualified itself by refusing to provide bounding box coordinates, apparently by design. Frankly, the author is unwilling to splurge >20€ just to check if the Pro version has a different system pre-prompt.

Otherwise, it’s worth pointing out that Claude, for our use cases, appears to have similar capabilities to a model such as Moondream.

ChatGPT

We’re using GPT-4o, as o1-preview doesn’t seem able to ingest images currently. For the demo image, using the prompt specified in the section intro, the model does provide a bounding box: (810, 370, 960, 520). The coordinates translate to the following clip:

Partially true, but not exact. The model seems to consider the distance readout as part of the designator. This quirk is not only present here but also in Claude and in Moondream. Honestly, not entirely sure what is the reason. The best guess is these sorts of UI elements, (i.e., designators) indeed often have supplementary icons or text "attached" to them, and this is reflected in whatever common data sets were used to train the models. So, for the model, the designator indeed comprises both the element itself and any "add-ons", an effect similar to the well-known "huskies are on snowy background" class of error – if it even can be considered an "error".

A follow-up prompt:

The bounding box provided is too large. It should only contain the designator’s square shape itself. Please refine the coordinates.

yields (842, 406, 927, 491):

better, but still not ideal.

Let’s move on to the busy image, where the bounding box is given as (617, 32, 1775, 1055):

Figure 19. Yes, this is the "bounding box" provided by ChatGPT for the "busy" image.

which, as we already guess from the coordinates alone, is way too imprecise – although, in fairness, it does include the designator itself. Funnily enough, the model, when requesting refinement, eventually "gives itself up" with this choice snippet:

It appears that my current attempt still did not yield the correct bounding box for the smaller red designator. Given the complexity of identifying the exact designator in this image, I would recommend a more manual analysis or specialized visual inspection using specific graphic tools. If you prefer, you could manually examine pixel clusters using an image editor to isolate the exact region you’re interested in.

All right, let’s do some follow-ups with "smarter" queries, like we did with Claude.

Figure 20. Result of the BB query on the "busy" image.

For the "BB location" prompt (the second one from the intro), ChatGPT seems to provide similar results as Moondream. Note that the image above was generated "manually", which is actually not strictly necessary – 4o absolutely can, and will, when asked for, draw the bounding boxes automatically.

Figure 21. Result of an object-counting query on the "busy" image.

Figure 22. Result of a similar query on the reference video frame.

ChatGPT returns similar results as Claude. When pressed with the suggestion of the correct answer (5), and asked to provide bounding boxes, it does so, but hallucinates heavily, and returns completely incorrect BBs^[13].

Figure 23. Yeah, no.

Summary

We have undergone quite a journey in this post. What then, are its takeaways?

The primary one, perhaps: for specialized use cases, such as, apparently, ours, there is still no free lunch. Most popular vision models are trained principally on "real world" images, with "real world" objects. No wonder – most of the use cases need those. For others, however, non-trivial work needs to be done, at least in the form of fine-tuning. Bummer on the one hand, yet on the other, a confirmation that a lot of interesting problem-solving is still there to explore.

The second takeaway flows from the first: even the models of the "large" corps, trained while burning through the equivalent electricity consumption of a small country (not an exaggeration), are unable to give answers to our problems for "free" – not because they are not deficient or borderline sci-fi at times, but they simply weren’t trained for this. And speaking of "free", were they even capable of performing the required tasks flawlessly, they’d still be prohibitively expensive for such a hobby use case – we’re talking dollars on the thousands of images/frames.

Finally, and most importantly, some conclusions relevant to the "offline" models we’ve gone through:

YOLO-World seemed to be the weakest for our use case, but even it can probably be made to work well with some fine-tuning, and not only for finding designators, but for mechs themselves. The latter, of course, presents a chicken-and-egg problem, as we need the data somehow.
Distilled VLMs such as Moondream work even better, and, perhaps in the near future, will be good enough to employ for the use case of detecting video-game-specific objects without any fine-tuning. We must, however, be mindful of their complexity and resource requirements, barely runnable on the majority of consumer-grade hardware. We cannot then unequivocally settle on them just yet.
Modern generalist segmentation models such as SAM 2 offer the most intriguing opportunity – sure, they’re not as convenient as auto-detecting target designators, but:
- they potentially output more precise annotation information via the masks;
- with a couple of days of work, the input data yield can surpass the autodetection approach, as we can locate and annotate arbitrary objects on the screen, not just the ones surrounded by a specific UI element;
- however, we will need to wait for less power-hungry derivatives for this to work in an acceptable timeframe.

Overall, we’re still not quite in the singularity that’s being heralded as coming soon for some time now, not even for "simple" tasks such as computer vision. But that’s good! It means there’s still a lot of interesting work to be done, and a lot of interesting problems to solve.

In the next entries, we will proceed with that problem-solving. We’ll be using several approaches to precisely extract the target designators, and we will start doing what we should have been doing already – comparing the performance and resource usage of various approaches in a diligent manner. We probably will even revisit some of the models (or their derivatives) discussed here. For now – until next time!

1. …except for getting the models to play nice with drivers, interfacing libs such as CUDA, and so on. War, war never changes…

2. See section 4.2 "Pre-Training" of the paper for the full collection of datasets.

3. There’s also the def column that can be wrangled with some basic NLP, but that is unlikely to bring any novel or contradicting conclusions.

4. It also manages that if you just provide "dropship" as a class, funnily enough.

5. And no, it also won’t detect a "mech" or "robot", at least in this screenshot.

6. Without paying for hardware worth as much as a good new car, that is.

7. From March this year, so already somewhat out of date, of course.

8. We could, of course, just use Pillow, as that’s what we’re starting with, but OpenCV’s API is slightly less cumbersome for the particular use case.

9. Not that generalized segmentation models or algorithms didn’t exist before – quite the opposite, of course. It’s just that the current crop is considerably more effective.

10. Ideally, for actual usage, the code should have been generalized to something like a Builder-pattern object. That would render the code a bit less digestible for the purpose of a blog-contained example, however.

11. Not counting the Rifleman II mech visible at the very end, crossing a gorge.

12. And the mech in question is an assault-class mech to boot.

13. For completeness, the prompt was: "The answer is actually 5 (five), not counting the player’s mech. Can you provide bounding box locations of these five mechs?"

Extracting Machine Learning Training Data from Video Games – UI element analysis for model training preparation

Mon, 10 Jun 2024 00:00:00 +0200

Figure 1. Illustration adapted from photo by George Pagan III on Unsplash

Intro

Welcome to the new entry on our series oriented around ML in video game use cases. The entry forms a part of a loosely connected series, the first of which sets up an overview of the problem space, the rationale, and the rough outline of how we’ll proceed. You can read it here if you want – but if you don’t, here’s a tl;dr:

we’re concerning ourselves with creating a situation-awareness-improving tool for a sci-fi, simulator-styled, multiplayer, First-Person Shooter, i.e., MechWarrior Online (MWO);
our task for now is to identify the position of the target designator on a frame (if present), and extract the image of the target contained within;
we’ll use this capability to extract training data (images of targets) to train our "final" model, which will serve as a tool for reviewing gameplay footage, for the purpose of improving situational awareness of players.

As a refresher, the following animation shows what the target designator looks like in the game (marked as 3.):

Your browser does not support the video tag.

and, in contrast, an example of an "actual", in-match screenshot:

Figure 2. In-match screenshot, "slightly" more busy. PPI has been obfuscated by box-bluring.

Our general process for the series can be broken down as follows:

we’ll first examine a number of different methods for extracting the designator’s position, including hyperparameter optimization, if applicable;
after that, we’ll run those methods on a larger dataset, comparing both their efficacy and efficiency.

Somewhere between 1 and 2, we’ll also need to develop a quality measure for extracted images, to minimize the amount of bogus or low-quality training data, such as one in the screenshot below.

Figure 3. An example of a bogus data point. Terrain completely obscures target’s image.

However, in this particular blog entry, we’ll focus on Point 0, namely: what makes a target designator in MWO? We’ll explore this question with the use of some basic Data Science methods.

Getting started

We’ll mostly be using OpenCV for frame loading and manipulation, as well as numpy for numeric operations on the frame’s data. Scikit-learn and other (deep) learning frameworks will come into play later.

This post assumes basic knowledge of all of the above (if you’d like an overview of OpenCV basics, here is one).

Having said that, let’s establish some standards we’ll be following in this and subsequent entries:

import cv2 as cv

# assuming we:
# - loaded frames from cv2.VideoCapture into a `video` array
# - the video in question is 1080p, and in color
frame = cv.cvtColor(video[frame_index], cv.COLOR_BGR2RGB)

print(frame.shape) # (1080, 1920, 3)
print(frame.type) # dtype('uint8')
print(frame.max()) # 255

In other "words", unless stated otherwise:

we’ll be using the RGB colorspace (after all, anyone using OpenCV for any length of time has an "amusing" story of accidentally using its BGR default);
our color values will be in the 0-255 range.

Designating the designator

We’re now looking for distinguishing features of the target designator, so that we can use that information to extract it from the input videos' frames in later, upcoming blog entries.

Fair warning about this section – we’ll be going pretty deep here, and sometimes into seemingly redundant paths. This is to show more possibilities than just the "optimal" solution for this particular case.

Going back to the matter at hand – looking at the last screenshot again, it becomes pretty obvious the distinctive quality of the target designator is its color. In fact, let’s see a couple more examples of the designators:

Figure 4. Designator examples in various situations, 1:1 size.

We can observe a couple of things:

the boxes are:
- of size 96px by 96px, at least in the input samples are using;
- in general, very red,
  - they are, however, not uniformly red, due to blending at the borders, video encoding particularities, and a slight translucency applied throughout;
the target designators can be, themselves, obscured by other UI elements, like the ring (arm) and crosshair (torso) reticles,
they are also not the only elements that appear to have this particular color – in two of the examples, we can see a third kind of reticle for lock-on weapons (no one said this game isn’t complex!) that, in its active state, has visually the same hue.

Let’s see how distinguishable that color is among the various reticles. The first kind of visualization tool that may come to mind is a histogram. We’ll use the following functions to generate them:

import numpy as np
import pandas as pd
import seaborn as sns

PIXEL_VALUE_LIMITS = (0, 255)


def to_channel_values_in_rows(image):
    channel_width = image.shape[-1]

    return np.moveaxis(image, len(image.shape)-1, 0).reshape([channel_width, -1])

def histogram_from_image(image, plt_axis, labels, colors, max_samples):
    channel_values_in_columns = np.transpose(to_channel_values_in_rows(image))

    hist_data = pd.DataFrame(data=channel_values_in_columns, columns=labels)

    sns.histplot(hist_data, ax=plt_axis, palette=colors, binwidth=10)
    plt_axis.set_ylim(0, max_samples) # necessary for consistency across all images

Instead of using matplotlib histograms, we’re going for seaborn’s version instead. This allows to more concisely define the graph parameters such as the colors and labels for each data element. We also need to extract the actual value frequencies from each color channel for the histogram to make sense – that’s where the to_channel_values_in_rows function comes in, converting the [y][x][channel] –> value mapping of the image into an array of dimension (channel_width, width*height), where every row lists the intensity values of pixels for the particular channels.

For an RGB histogram, we invoke the function like so:

histogram_from_image(image, axs_from_matplotlib, ["r", "g", "b"], ["r", "g", "b"], 96**2 * 0.5)

The max_samples is derived from the size of the image (target designator size), but ultimately something obtained via trail-and-error.

OK, let’s see what we got:

Figure 5. Sample target designators with corresponding RGB histograms.

That’s not… very helpful, is it? The values are seemingly all over the place, we can mayyyybe make out a small bump in the R-channel’s values around 200, but that’s it.

We mustn’t give up on histograms quite yet, 'though. RGB is not the only colorspace. Alternatives include HSL and HSV colorspaces that, as the linked Wikipedia page states, align more closely with human visual perception than RGB.

Onto the histograms:

histogram_from_image(cv.cvtColor(image, cv.COLOR_RGB2HLS), axs_from_matplotlib, ["H", "L", "S"], ["black", "magenta", "cyan"], 96**2 * 0.5)

Figure 6. Same target designators with their HSL histograms

Immediately, we see that all diagrams have a distinctive peak in the Hue channel, within the 170-180 bin. So, what does, say, 175 at max saturation and half lightness (both for better color visibility) looks like? Like this:

SWATCH_IMAGE_SIZE = 40
COMPONENT_VALUE_MAX = PIXEL_VALUE_LIMITS[1]  # 255 in our case


def display_color_swatch(h, l, s, image_size=SWATCH_IMAGE_SIZE):
    demo_image = np.tile([h, l, s], (image_size, image_size, 1)).astype('uint8')

    # housekeeping to ensure image is displayed in 1:1 ratio
    figsize(image_size * px, image_size * px)
    plt.axis('off')
    plt.tight_layout(pad=0)

    plt.imshow(cv.cvtColor(demo_image, cv.COLOR_HLS2RGB))

    plt.show()


h = 175
s = COMPONENT_VALUE_MAX
l = COMPONENT_VALUE_MAX / 2

display_color_swatch(h, l, s)

Yeah, that does look red all right.

In the previous couple of snippets, we’ve been using HLS (instead of HSL) since that what OpenCV offers (similarly to BGR vs RGB). Keep that in mind, so as not to mix up the channels.

For good measure, let’s take a portion of each of the boxes – in this case, the "upper-left" one, i.e. image[8:37, 7:10, :], and generate the histograms for that:

This indicates even more strongly that just going by hue value might be our ticket (since the same "spike" is visible right around the 170-180 bin).

To preempt the eventuality that masking by a single channel value might be insufficient, we can examine the relationship between the different channels. At first, visually. We need a 3D scatter plot. Our function to generate one looks like this:

AX3D_PREFIX_SETTER = "set_"
AX3D_AXES = ["x", "y", "z"]
AX3D_LIM_SUFFIX = "lim3d"
AX3D_LABEL_SUFFIX = "label"


def set_up_3d_plot_limits_and_labels(ax, labels: tuple[str, str, str]):
    """Helper function - sets all axes limits and labels"""

    def __ax3d_funcs(ax3d, suffix):
        return [getattr(ax3d, f"{AX3D_PREFIX_SETTER}{axis}{suffix}") for axis in AX3D_AXES]

    for limit_setter in __ax3d_funcs(ax, AX3D_LIM_SUFFIX):
        limit_setter(*PIXEL_VALUE_LIMITS)

    for label_setter, label in zip(__ax3d_funcs(ax, AX3D_LABEL_SUFFIX), labels):
        label_setter(label)


def scatter_3d_from_image(image, labels, fig, plt_axis, num_columns, num_rows, column, row):
    channel_values_in_rows = to_channel_values_in_rows(image)

    plt_axis.axis('off')
    ax = fig.add_subplot(num_rows, num_columns, (row * num_columns) + column + 1, projection="3d")
    set_up_3d_plot_limits_and_labels(ax, labels)

    ax.scatter(*channel_values_in_rows)

And our invocation, for example, for the 2nd row, and the 3rd image, might look like this:

fig, axs = plt.subplots(3, 10)

row = 1
image_index = 2

# Some code in between...

scatter_3d_from_image(frame, ["r", "g", "b"], fig, axs[1, 2], 10, 3, 2, 1)

The code for the current visualization appears convoluted. That’s because we’re mixing up 2D and 3D diagrams in one plot. The 3D elements force us to use the fig.add_subplot API, whereas our 2D elements are reliant on the plt.subplots API, with array of 2D axes. This is why we also include the plt_axis.axis('off') call – we need to "hide" the axes of the 2D diagram "slot" we’ve created with plt.subplots.

Right, let’s see our results:

Figure 7. Scatter plots for target designator "corner fragment". First row is the 28px*3px image, scaled up, second is the RGB scatter plot, third is HLS scatter plot.

Figure 8. Scatter plots for complete target designators. First row is the 96px*96px image, scaled up, second is the RGB scatter plot, third is HLS scatter plot.

For the fragment version, the clustering across the two different colorspaces is pretty similar — arguably, the HLS one looks more "compact", but that might be misleading. The full designator versions offer a starker difference between RGB and HLS. In most cases, in the HLS plot, we can make out the same kind of cluster as in the fragment diagrams, whereas the RGB versions are much more of a chaotic jumble.

However, it is high time we started to act like true MechWarriors – in this case, stop relying on the MK. I Eyeball, and turn to cold and calculating machine systems for our target acquisition. This means, in our case, clustering.

What we’ll do now is join both imagesets into large images, and run a clustering algorithm on them. This will "smoothen out" differences across our samples and hopefully make the cluster we’re interested in - the target designator color cluster – more pronounced, and thus easier to pick out by the algo.

Speaking of algos, scikit-learn has a very convenient overview of the clustering algorithms it offers. Now, we need to consider our requirements and preferences.

As far as requirements are concerned, anything with "even cluster sizes" is right out. Not only it’s unlikely to be the case just looking at the variability of colorspace values across our images, but also we’re really interested in the one (postulated) cluster that will represent the target designator colorspace values. So K-Means, Spectral clustering and Bisecting K-Means should be excluded.

For scaling, we don’t really about it – our data space size is too small for it matter significantly.

We now have a choice of two broad categories of algos, split on the kind of main parameter, that being:

distance threshold: this is… kind of what we want to obtain from the clustering. We can have a good guess, but let’s maybe leave algos requiring in case something doesn’t work.
neighborhood size: also arguably something that we want to get from the clustering, but in this case we can at least estimate some minima from the size of the box elements. So, let’s go with that first.

This leaves us with Affinity propagation, DBSCAN, OPTICS, Gaussian mixtures, and BIRCH. We can start with any one of them. However, DBSCAN looks like the best candidate so far. That’s because of a quirk in its parametrization. To quote the docs:

[…] the parameter eps is crucial to choose appropriately for the data set and distance function and usually cannot be left at the default value. It controls the local neighborhood of the points. When chosen too small, most data will not be clustered at all (and labeled as -1 for “noise”).

This low amount of clustering and labeling most data as "noise" is, in our case, exactly what we want!

Let’s get to work then. First, let’s merge all our images into one – we can do this with a NumPy array-level operation:

# all images
designator_images: list[np.array] = ...

# all with just the designator segments
designator_segment_images = [img[8:37, 7:10, :] for img in designator_images]

designator_image_concat = np.concatenate(designator_images, axis=1)

designator_segment_images_concat = np.concatenate(designator_segment_images, axis=1)

This will produce the following two images:

Let’s start with the latter first, as the result obtained from that will help us find the "right" cluster of interest in the former. In fact, because this image contains just (a portion) of the designator, we want the clustering algorithm to generate exactly 1 cluster, and leave the rest as noise.

Alright, so DBSCAN has several parameters, of which two are of particular interest: eps and min_samples. Both determine how the neighborhood of "core" points is defined – an important distinction, by the way, between that and the size of the entire cluster, which DBSCAN does not concern itself with directly.

Keeping this sizing caveat in mind, we’ll ballpark both parameters.

For eps, we want a decent, but not to overly broad of a distance, so that the cluster doesn’t capture too many points. A good value would be to at least allow a distance of 2 in any direction (H, S, L). Since we’re keeping the default Euclidean metric, this gives us eps=2**3=8. Of course, this also lets through points that are 8 values away along any one axis, but this won’t be a problem here.

Now, min_samples. One would be tempted to get a large number, like the size of a single of one of the 3 "bars" each designator segment has (recall the individual images are of shape (3, 28) in this case). This would give us 28*10=280 (10 being the image count). Great! Except it won’t work – no cluster will be recognized. No "core" point can apparently be found for the given eps value in this case. However, half of that, i.e. 140 works, so let go with that.

Our clustering result is generated through the following code:

from sklearn.cluster import DBSCAN

def prepare_image_for_clustering(image):
    # since we've determined HSL/HLS to likely be superior for clustering
    hsl = cv.cvtColor(image, cv.COLOR_RGB2HLS)

    # DBSCAN, as typical of sklearn Estimators, needs a columnar format of the values
    in_rows = to_channel_values_in_rows(hsl)
    return np.transpose(in_rows)


def hsl_clusters_of(X, eps, min_samples):
    # we're explicitly adding that we want a l2/Euclidean metric
    dbscan = DBSCAN(eps=eps, min_samples=min_samples, p=2)

    # our clustering fit
    return dbscan.fit_predict(X)


X_segment = prepare_image_for_clustering(designator_segment_images_concat)

clusters = hsl_clusters_of(X_segment, eps=2 ** 3, min_samples=(28 * 10) // 2)

clusters is simply a 1-D denoting which cluster a given sample belongs to. We’ll fit it into a dataframe to allow for analysis and display^[1] :

COLORSPACE = ("H", "L", "S")


def cluster_to_df(image_data_h_l_s, clusters):
    return pd.DataFrame(data=zip(image_data_h_l_s[:, 0], image_data_h_l_s[:, 1], image_data_h_l_s[:, 2], clusters), columns=list(COLORSPACE)+["cluster_id"])


clustering_df = cluster_to_df(X_segment, clusters)

and start verifying that we got our desired result:

clustering_df["cluster_id"].value_counts()

# prints out:
#  -1    480
#  0    390
# Name: cluster_id, dtype: int64

Yup, we’ve got our single cluster (since -1 is the special "noise" value). Now for the cluster’s summary statistics:

target_values = clustering_df[clustering_df["cluster_id"] == 0]

target_values.describe()

# prints out:
#                 H           L           S  cluster_id
# count  390.000000  390.000000  390.000000       390.0
# mean   172.815385  101.858974  254.423077         0.0
# std      1.672942    8.157392    1.863733         0.0
# min    170.000000   83.000000  243.000000         0.0
# 25%    172.000000   96.000000  255.000000         0.0
# 50%    173.000000  102.000000  255.000000         0.0
# 75%    174.000000  109.000000  255.000000         0.0
# max    179.000000  121.000000  255.000000         0.0

The mean hue value is close to what we estimated earlier from the histograms.

Now, let’s try to run the clustering on the concatenated full designator images. We’re going to multiply min_samples by 4, as every image has that number of previously extracted segments.

X_designators = prepare_image_for_clustering(designator_image_concat)

clusters_full = hsl_clusters_of(X_designators, eps=2 ** 3, min_samples=((28 * 10) // 2)*4)

clustering_df_full = cluster_to_df(X_designators, clusters_full)

clustering_df_full["cluster_id"].value_counts()

# prints out:
# -1     61136
#  1      7300
#  2      4137
#  5      3869
#  7      3185
#  4      2553
#  3      2082
#  9      1989
#  6      1886
#  0      1795
#  8      1597
#  10      631

Lots more clusters, but that’s to be expected. The first cluster is hopefully want we want, followed by, most likely, greyscale values in some of the test images. Not leaving anything to chance, let’s check out what hue values are represented by each cluster.

We’re going to calculate three percentiles of the hue values for each cluster: the 10th, the 50th (i.e., median), and the 90th. This is a serviceable exploration heuristic if a quick check is desired, and a relatively varied distribution is suspected. We’re also including the cluster size again, for good measure.

from functools import partial


# The keywords of agg are completely arbitrary - they're just
# our column names in the output. What is important is that we provide
# a function object (Callable) for each value. That is why we need to
# invoke partial for the percentiles.
clustering_df_full.groupby(by="cluster_id")["H"].agg(centile_10=partial(np.percentile, q=10),
                                                     median=np.median,
                                                     centile_90=partial(np.percentile, q=90),
                                                     count=len)


# prints out:
#             centile_10  median  centile_90  count
# cluster_id
# -1                 7.0    30.0       173.0  61136
#  0                 0.0     0.0         0.0   1795
#  1               171.0   173.0       177.0   7300
#  2                 0.0     0.0         0.0   4137
#  3                 0.0     5.0        13.0   2082
#  4                30.0    30.0        30.0   2553
#  5                15.0    26.0        34.0   3869
#  6                 7.0    16.0        20.0   1886
#  7                27.0    31.0        33.0   3185
#  8               165.0   165.0       168.0   1597
#  9                 0.0     0.0         0.0   1989
#  10                0.0     0.0         0.0    631

(by the way: yes, we could have just used describe here – the only benefit is a slightly more focused output. Don’t worry, we’ll come back to that method later on.)

Phew, looks like we do have most of the relevant values in one cluster (cluster_id == 1). The guess that the other of the largest clusters represent greyscale values was also correct (if you’re wondering why there are multiple ones with H values 0 – that’s because they almost certainly differ in the other colorspace components).

The only mildly worrying thing is cluster 8, being very close in hue to the red of our designator. We’ll keep that in mind as we progress into the next steps.

Before we close this section, it would serve us to actually visualize the cluster spaces. We’re going to do it in two ways – one, by using a predefined colormap, the other, by using the actual (median) colors of the cluster. The code for the diagram generation is as follows (warning – lots of matplotlib idiosyncrasies we won’t go into detail here):

import matplotlib.patches as mpatches

MISSING_CLUSTER_ID = -1
MISSING_CLUSTER_COLOR = [0.75, 0, 0.75]

LEGEND_VALUES_PER_COL_MAX = 2


def show_legend_no_alpha(ax, colors, labels, title):
    """Helper function - displays a legend for the cluster colors with 0 transparency"""
    # removing the alpha channel (RGBA -> RGB)
    handles_colors = [c[:3] for c in colors]

    # following
    # https://matplotlib.org/stable/tutorials/intermediate/legend_guide.html#creating-artists-specifically-for-adding-to-the-legend-aka-proxy-artists
    final_handles = [mpatches.Patch(color=color, label=label) for (color, label) in zip(handles_colors, labels)]

    # and ensure that the cluster values are the labels...
    ax.legend(handles=final_handles, ncols=(len(final_handles) // LEGEND_VALUES_PER_COL_MAX) + 1, loc="upper right", title=title)


def draw_clusters_cm(cluster_data: pd.DataFrame, alpha: float = 1., colormap="tab20b"):
    """Primary presentation function – draws provided clusters with a predefined colormap
    See https://matplotlib.org/stable/tutorials/colors/colormaps.html
    """
    fig = plt.figure()
    ax = fig.add_subplot(projection='3d')

    set_up_3d_plot_limits_and_labels(ax, COLORSPACE)

    scatter_plot = ax.scatter(*[cluster_data[comp] for comp in COLORSPACE],
                              c=cluster_data["cluster_id"],
                              alpha=alpha,
                              cmap=colormap)

    handles, labels = scatter_plot.legend_elements()
    show_legend_no_alpha(ax, [c.get_color() for c in handles], labels, "cluster_id")

    plt.show()


def draw_clusters_real_color(cluster_data: pd.DataFrame, alpha: float = 1.):
    """Alternative presentation function – draws provided clusters with "real" colors,
     i.e. each of the median HLS values for the cluster."""
    fig = plt.figure()
    ax = fig.add_subplot(projection='3d')

    set_up_3d_plot_limits_and_labels(ax, COLORSPACE)

    # create a color dict for the cluster colors
    median_cluster_colors = cluster_data.groupby("cluster_id")[list(COLORSPACE)].median()

    median_cluster_color_rgb = cv.cvtColor(np.array([median_cluster_colors.values]).astype('uint8'), cv.COLOR_HLS2RGB)

    colors_per_cluster = dict(
        list(zip(median_cluster_colors.index,
                 (median_cluster_color_rgb[0] / float(PIXEL_VALUE_LIMITS[1])).tolist())))

    # let's set -1 to an unusual color, but background color (such as light magenta) for good measure
    if MISSING_CLUSTER_ID in colors_per_cluster:
        colors_per_cluster[MISSING_CLUSTER_ID] = MISSING_CLUSTER_COLOR

    # we're doing things differently now - drawing individual scatter plots per cluster
    for cluster_id, color in colors_per_cluster.items():
        cluster_data_specific = cluster_data[cluster_data["cluster_id"] == cluster_id]
        ax.scatter(*[cluster_data_specific[comp] for comp in COLORSPACE],
                   color=color,
                   label=cluster_id,
                   alpha=alpha,
                   )

    handles, labels = ax.get_legend_handles_labels()
    show_legend_no_alpha(ax, [h.get_edgecolor()[0] for h in handles], labels, "cluster_id")

    plt.show()

and these are the diagrams we want to generate:

draw_clusters_real_color(clustering_df[clustering_df["cluster_id"] == 0])

draw_clusters_cm(clustering_df_full[clustering_df_full["cluster_id"] != -1])
draw_clusters_real_color(clustering_df_full[clustering_df_full["cluster_id"] != -1])
draw_clusters_real_color(clustering_df_full, 0.01)

which gives us the following:

Figure 9. Scatter plots for (going left to right): 1) The sole cluster in the segment image. 2) Clusters in the full image, shown with their median colors. 3) Clusters in the full image, shown with a predefined colormap/palette. 4) All pixels in the full image, including non-clustered ones.

Couple of notes here:

we were needlessly concerned about cluster 8 being, perhaps, cluster 1 's "lost twin" – from the diagram, especially the median color diagram, it is apparent that this cluster is actually very far away in our colorspace. Just shows the importance of properly presenting data and information extracted therefrom, and not jumping into conclusion based on solely a partial view of the data.
speaking of visualisation:
- compare diagram 2 with diagram 3 – while the "true color" diagram provides some immediate information as to what the clusters represent, only the "palette" really demonstrates cluster boundaries.
- for the final diagram, the alpha of all points is reduced greatly. This is because the no-cluster classification dominates within our dataset, so using no alpha would render our diagram pretty much unreadable.

And now, for the pièce de résistance – color information for cluster 1 in the full image!

Or, well, it would be, but we need to take of one thing first. Some readers have probably noticed the way we calculated the median color values of the cluster, i.e., get the median of each individual HSL component. That grouping of values is actually a marginal median. It is not the sole representation of a median in multidimensional spaces. In fact, depending on your dataset and on the relations between the components, it may be completely non-representative of the examined data, as discussed in this answer on Cross Validated.

At the surface level, the Cluster Of Interest looks like your typical unassuming, convex blob with likely straightforward relationships between the components. This may be deceiving, as, just by looking at the diagram, we cannot really see what the actual value densities are within that cluster. So, let’s go one step further and make sure we get our values right.

Another, somewhat more generally robust median in a multidimensional space is the geometric median^[2]. It’s not available out-of-the-box in Pandas or numpy – but there is a third-party library that provides it.

To calculate it, we first need to extract the combined HLS values into a dedicated column:

# filtering out just the cluster of interest
cluster_1_data = clustering_df_full[clustering_df_full["cluster_id"] == 1].drop(columns=['cluster_id'])


# extracting the combined values
cluster_1_data["HLS"] = [r for r in cluster_1_data[["H", "L", "S"]].values]

# normalizing them to [0,1], as needed for the median calculation
# doing so in a separate step means we have access to the vectorized operation syntax
cluster_1_data["HLS_norm"] = cluster_1_data["HLS"]/COMPONENT_VALUE_MAX

# keeping just the normalized combined column
cluster_1_data = cluster_1_data.drop(columns=["HLS"])

Now we can actually obtain the geometric median:

from geom_median.numpy import compute_geometric_median

def compute_geom_median_on_series(series: pd.Series):
    result = compute_geometric_median(series.values)
    # "denormalize" the result back to [0, 255]
    return result.median*COMPONENT_VALUE_MAX

geometric_median = cluster_1_data["HLS_norm"].agg(compute_geom_median_on_series)

geometric_median

# prints out
# array([173.03971593, 100.6201285 , 254.43690908])

Now to demonstrate it, using the color swatch function we’ve defined previously…

# "unwinding" the result array into positional arguments
# important to keep the sequence right, which in this case
# it is (H,L,S)
display_color_swatch(*geometric_median)

…and compare to the individual medians:

color_components = ["H", "L", "S"]
cluster_1_data[color_components].agg(np.median)

# prints out
# H    173.0
# L    101.0
# S    255.0
# dtype: float64

We can see that, while in this case, the result is almost identical, there is still a small difference (in the saturation).

For good measure, let’s also generate more summary statistics for the color components, this time using describe:

color_components = ["H", "L", "S"]
cluster_1_stats = cluster_1_data[color_components].describe(percentiles=[0.1, 0.5, 0.9]).round(2)

cluster_1_stats

# prints out
#              H        L        S
# count  7300.00  7300.00  7300.00
# mean    173.59    96.76   253.86
# std       2.39    15.72     3.06
# min     168.00    55.00   240.00
# 10%     171.00    72.00   250.00
# 50%     173.00   101.00   255.00
# 90%     177.00   114.00   255.00
# max     180.00   127.00   255.00

Finally, let’s summarize our results into a single DataFrame:

cluster_1_stats_final = cluster_1_stats.copy()

cluster_1_stats_final.loc["geom_median"] = geometric_median

cluster_1_stats_final = cluster_1_stats_final.drop(index=["count"]).round(2)

cluster_1_stats_final

# prints out
#                   H       L       S
# mean         173.59   96.76  253.86
# std            2.39   15.72    3.06
# min          168.00   55.00  240.00
# 10%          171.00   72.00  250.00
# 50%          173.00  101.00  255.00
# 90%          177.00  114.00  255.00
# max          180.00  127.00  255.00
# geom_median  173.04  100.62  254.44

Summary

The numerical values we’ve now obtained will help us gauge the extent of the filtering criteria, as we move on to actually extracting the designators themselves.

Now, was this much work necessary to determine the colors that interest us? The answer is "absolutely not".

We could’ve just put the example images into an image editor and let it sample the color. Then, we might’ve eyeballed the color component intervals (i.e., min, max, and so on) for the purpose of creating a prototype extractor of the designator images. This is, in fact, the level of effort that should usually be applied when making prototypes.

Hell, we could’ve gone with an alternative route (if allowed by relevant copyright law and additional agreements such as the EULA) – look into the game’s graphic assets, if available, and find the elements from which the designator is constructed. This would allow us to estimate the color range from stuff like the designator’s alpha channel^[3].

Moreover, our data size is suspiciously small. In a project of similar magnitude, we should have hundreds, if not thousands of samples to use for the clustering. The reliability of the result rests pretty much on the author’s confidence to handpick representative samples consistently.

The intent of this post, however, was twofold.

First, to demonstrate how to go about solving this kind of problem somewhat more rigorously, so that we can have a little bit more confidence in what we base our further work on.

The other was showcasing a number of data analysis tools and the way to use them, allowing them to be used when tackling similar data extraction tasks.

Well, here we are. In the next couple of entries, we’ll proceed with the image extraction itself, using several different techniques of varying complexity, power, and performance. To provide a bit of a teaser: in the immediately subsequent post, we’ll start with a couple of simple and current techniques, including the usage of some libraries that are definitely more modern than OpenCV. Watch this space!

1. If you need a general refresher on what Pandas DataFrames are, have a look at this blog post.

2. Do not confuse it with the geometric mean. The geometric median deals directly with spatial relationships of the values, whereas the "geometric" in the geometric mean refers to the distribution of values in some set of real numbers.

3. If it’s not set programmatically, that is.

Extracting Machine Learning Training Data from Video Games – Problem Statement

Sun, 5 May 2024 00:00:00 +0200

Figure 1. Source: SD 2.1+own work.

Spring has sprung in full force in the Northern hemisphere, so it’s time to dust off the old "submit publication" button! In this entry, we’ll look into ways of extracting training data from computer-generated data sources without breaking too much of a sweat. Specifically, we’ll introduce the data source we’ll work on, and define a problem statement. This will form a basis for later blog entries that will tackle various aspects of said problem. Read on!

Introduction

One of the interesting aspects of our times is the increasing importance of data derived from "human-made" sources. Early automation was almost always primarily grounded in the physical reality. As time went by, systems that built upon data delivered by other systems become more worthwhile: analysis of social media trends, electronic sales projections, and so on. And, well, there’s also lots of stuff built on top of data originating from video games. Anti-cheat systems, usage analytics – used both for objectively valuable insights like improving accessibility, or less so, like "optimizing" microtransactions – with the normalization of gaming as a pastime for all ages, there’s a gold mine of opportunity here. And we’ll take some advantage of it; but first, a bit more about our subject.

The game in question is MechWarrior Online, subsequently referred to as MWO. As the name suggests, it is a multiplayer-only fair set in the Battletech/MechWarrior universe. It offers several game modes, virtually all of them centered around two teams fighting each other to destruction, while also trying to balance achieving auxiliary objectives.

Figure 2. Promotional screenshot of MechWarrior Online. An Ebon Jaguar heavy mech engaging a target (possibly an armless Bushwacker) with support weapons. Source: Piranha Games.

Everyone pilots a Battlemech (no combined arms here) : a customizable, heavily armed, ludicrously armored, single-seat, bipedal^[1] combat vehicle. While it sounds like a First-Person Shooter with some sci-fi bling thrown on, that couldn’t be further from the truth. Mechs have multiple, independently destroyable components, each potentially housing multiple armaments and pieces of equipment. Players need to manage ammunition, heat generated by own (or hostile^[2]) weapon fire, damage distribution, terrain use, team coordination, and so on.

The general vibe feels less like a squad-based FPS, and more like somewhere between naval combat and simultaneously controlling an armored platoon. Mad aiming skillz are much less important here than situational awareness, forethought, planning, and team coordination. Not only that, but the mechs themselves are – as previously stated – highly customizable, to the point that a significant amount of player’s time is spent tweaking and trying out different weapon/equipment configurations. All this coalesces into a unique experience, and so a unique data landscape to work on.

Possible premises and their choice

Having explained the circumstances we have on our hands, let’s see what we can do with the source material. Since many players occasionally record their games for later analysis (and some for streaming), the original idea was to create a "virtual coach" that would, based on said recordings, call out possible improvements in the player’s style.

Positioning (w.r.t. friendlies and the likely enemy placement) is probably the most important learnable skill in MWO - but getting full data for that (known friendly and enemy positions at a given point of time) is difficult to extract just from existing footage. Not only that, developing the model itself would be decidedly non-trivial. In other words – assisting positioning in MWO is an intriguing challenge, but complex enough to warrant a whole separate series.

So – at least for this blog series – let’s try something more manageable: situational awareness. With all the aspects of managing a mech occupying the player brain’s processing power, slipping up and being completely oblivious to a hostile mech^[3] running through the field of vision is surprisingly common. However, such mistakes can usually prove fatal, as said overlooked opponent can easily get behind the player’s mech and start tearing them apart. Moreover, the initial situation often constitutes the opponent’s error, and would be prime time to engage.

Having contextualized our circumstances, we now need a problem statement with specific requirements and goal conditions. Here it is:

Develop a model that detects enemies in-view that could have been targeted, but weren’t, and mark them on the footage. Bonus goal: mark situations where the player was not actively engaging any target, but could.

OK, looks good and relatively manageable. From our problem statement, we can work out that we need some way of determining:

the positions of mechs in a given frame of the footage;
that a given mech is friendly or hostile;
that a given hostile mech is "non-targeted";
that the player is in a "non-engaging" state.

We can also declare a couple of non-functional requirements:

our solution does not have to be especially performant – we’re operating on pre-recorded footage;
our solution should weigh recall over precision — we’re fine if e.g., a friendly mech is falsely marked as a "non-targeted" hostile mech, as it should be apparent from the context of the footage, and the recommendation simply discarded.

Now, we’ll take a brief look at the game’s UI to determine what, exactly, we want to train.

Identifying screen UI elements relevant for machine learning model training

Let’s examine some screenshots, enhanced with context markings. Examples are demonstrated in the slideshow below:

Your browser does not support the video tag.

Targeting reticles: yes, the mechs have multiples thereof, for different types of weapons, and their hardpoint locations.
Unmarked hostile mech.
Target designator on a marked, hostile mech: the player can mark at most one hostile mech at a time.
Targeted mech’s scan result, showing weapon and component status^[4].

This is just some information relevant to the player on the screen, but pretty much all that we need for our purposes.

In end effect, we have two types of data to extract from the video’s frames:

UI-derived information, such as weapon firing state,
detectable objects.

While the former is extractable using simple image manipulation and "classic" computer vision techniques, this is not so with detectable objects, i.e. the mechs. For that, we need to train some sort of object detection model.

We could go over each recording and meticulously mark each and every mech. But who has the time (or money to hire annotators) for that?

We might consider "traditional" motion detection techniques, used widely in consumer IP cameras (and explained in a myriad of online tutorials), but that option also falls flat. Why? Because both the objects and the camera are moving – sometimes quite vigorously. So that’s one possible free lunch out of reach. We will, however, consider the potential to exploit research into movement detection on mobile cameras, but that’ll come later on.

Now, take another look at the screenshot: see how the hostile mech is nicely marked^[5]? And how about that nice bracketing of the actively targeted mech? Almost like a bounding box, right?

Well, it looks like we have a way out – we’ll try to automatically extract detection boxes by annotating targeted hostile mechs as objects to be detected. We can use that data as inputs for subsequent training of our "primary" detection models.

Figure 3. For completeness, a screenshot that’s more representative of situations in the game – an evidently more complex scene. The view is unzoomed, meaning the full interface, including the quasi-diegetic cockpit displays, is visible. The player, and two teammates, are engaging an (unlucky) hostile mech that just rounded a corner. The opponent’s readout is showing damage to the torso components, including actively receiving fire to the central portion. PPI has been edited out.

Summary

In this entry:

we’ve introduced the use case we’re going to handle in the blog series initiated by this entry: extracting training data from video games, and putting it to use.
We’ve chosen MechWarrior Online (MWO) as our exemplary data source.
We’ve also examined the automation problem landscape in MWO:
- we considered several potentials, such as assisting in player positioning,
- but, for the immediate future, settled on a more manageable problem: helping with situational awareness.
Finally, we also identified the screen elements that are relevant for our model training.

In the upcoming several follow-up entries of the series, we’ll explore how to obtain training data for the defined task, by way of identifying and extracting the relevant UI elements. We’re going to use, and eventually compare, several different methods to accomplish this task. And yes, that means we’ll actually start writing some code. Stay tuned!

1. Battletech fans will be quick to note that the last two points aren’t always the case in the setting, but it is in MWO, so you can safely ignore them (unless they’re in your rear arc).

2. …or sometimes friendly…

3. Especially a small and fast one that is equipped for stealth.

4. This one in particular is at full health.

5. That’s the weight class marker. The cross-hatched diamond signifies the "assault" class, the heaviest one in MWO.

Combining Python assignment expressions and destructuring: Having your variable and eating it too

Sun, 6 Aug 2023 00:00:00 +0200

Intro

One of Python’s probably most "wow, this is great!" features for newcomers is destructuring in assignments, whether of tuples, lists, or – indeed – any iterables:

destructuring_intro.py

a, b = ("a", "b")
[zero, one, two] = range(3)

For the longest time, this destructuring, and for comprehensions, were, sadly, all that Python had to offer in terms "structurally-oriented" syntactic sugar. In contrast, Scala’s pattern matching, while arriving a bit later in the game, has always been more feature-packed and functional.

Fortunately for Python, it has been eventually extended with structural pattern matching, which brings it way closer to the expression power of Scala’s pattern matching. "Almost" for several reason, the largest being that structural pattern matches are statements, not expressions. Why this was done was actually addressed in the PEP here; personally, the argument, possibly paraphrased as "it wouldn’t be Pythonic" is as lost on me as the opportunity that was available here.

In any case… this post is not about structural patching matching, but closing another type of expression power gap between Python’s and Scala’s destructuring semantics, also becoming a reality relatively recently: with Python 3.8.

By the way: this is definitely a shorter entry, written simply because I stumbled upon the relevant problem (and solution) while writing a considerably longer series of blogs. Hopefully, they’ll start appearing this quarter – but until then, let me show you something…

Schrödinger’s destructuring

We’ll start with an example in Scala^[1]. Let’s say we have a three-element list, and we want to assign said elements into separate variables. We end up with something like this:

pattern_matching_simple.scala

val List(one, two, three) = List(1,2,3)

The code above maps easily to Python, with the addition of the val keyword (for final, i.e. non-reassignable variables) and the syntax of defining a list being the only two differences. Indeed, the corresponding Python code looks like this:

[one, two, three] = [1, 2, 3]

OK, let’s say we know want to store both the individual elements and the entire list. In Scala, this is done easily enough with the addition of an @ assignment:

pattern_matching_whole.scala

val result@List(one, two, three) = List(1,2,3)

// In the console, this prints out:
//
// result: List[Int] = List(1, 2, 3)
// one: Int = 1
// two: Int = 2
// three: Int = 3

So what about Python? Until a couple of years ago, all one could do is something like this:

result = [1, 2, 3]
[one, two, three] = result

Not exactly the end of the world, but a bit annoying.

Things have changed with the introduction of assignment expressions, which you may be familiar with, by way of :=, i.e. the walrus operator^[2]. As the name suggests, these are simply assignments (=) that are also expressions – meaning they evaluate to some value, unlike the standard assignment statement.

Let’s have a first go at trying to use an assignment expression in our test case:

result = ([one, two, three] := [1, 2, 3])

The snippet emulates Scala’s ordering of assignments, and looks like it could work, except the interpreter prints out:

  File "<stdin>", line 1
    result = ([one, two, three] := [1, 2, 3])
              ^^^^^^^^^^^^^^^
SyntaxError: cannot use assignment expressions with list

The same (or similar) gets printed out for tuples, sets, and so on. Using destructuring in an assignment expression is simply not supported. Not all is lost, however. All we have to do is reverse the sequence of assignments:

pattern_matching_whole.py

[one, two, three] = (result := [1, 2, 3])

And this works! Here’s the corresponding, verifying console output:

>>> one
1
>>> two
2
>>> three
3
>>> result
[1, 2, 3]

Collapsing the wave function

To summarize – for a given value, what we wanted is to assign:

that full value, and
a destructured version of the value,
in one step.

Our solution boils down to using an assignment expression together with good-old destructuring, represented by the general pattern below:

destructure_assign_pattern.py

<destructuring_vars> = (<whole_value_vars> := <value_to_destructure>)

Again, this is possible in any version of Python starting from 3.8, inclusive.

One important remark is that, like in the "old" two-liner version, we actually reference the original value within the assignment expression. Said identity has consequences when the value is mutable, such as the list in the example below:

destructure_assign_gotcha.py

org_value = [1, 2, 3]
[one, two, three] = (result := org_value)
result.append(4)
print(org_value)
# prints:
# [1, 2, 3, 4]

This might be a disadvantage when rewriting code where, originally, the "whole result" is reconstructed from the destructured, constituent parts, like so:

destructure_assign_reconstruct.py

[one, two, three] = [1, 2, 3]
result = [one, two, three]

creating a brand-new list. In other words, be wary of the quirk when rewriting code to this pattern.

Now, to get to a final talking point, one that’s probably on a number of the readers' heads: should this kind of "trick"/pattern be used?

For starters, I am definitely not a "pure" Python programmer (nor, TBH, do I aim to be one), and so no authority of whether something is "Pythonic" or not.

Otherwise, it can be argued that this pattern sacrifices readability for the sake of terseness – but the same could be said about any use of the assignment expression. The syntax in question is indeed a very sharp tool in the developer’s drawer, able to muddle up the codebase if abused. However, it still exists for a reason, and that reason is that sometimes these "shortcuts" do improve readability by increasing proximity of the assignment to other, relevant parts of the code.

For me specifically, the need arose when I was transforming OpenCV’s bounding boxes. The BBs are encoded as a 4-tuple^[3]; I had a scenario where I needed to process both meaningfully-named, individual components of the BB, and the entire BB tuple.

The function in question contained less than 10 LoCs, so the call, in my mind, was simple. And this is probably the answer to the question posited above: when following general software engineering practices (including considering your team’s style and abilities), it’s OK to use the pattern whenever necessary and practical.

Hope this window into the possibilities offered by Python’s expanding syntax will prove useful to at least some readers. Happy coding!

PS. To be completely clear: I categorically do not claim "discovery" of this pattern, and I frankly strongly doubt I was the first to describe it. I simply haven’t seen – or don’t remember seeing – it being discussed anywhere, hence this blog entry :).

1. Don’t worry, the examples in Scala will be simple and completely explained.

2. To finalize my rant from the intro, note how this PEP – introducing a brand-new way of "converting" what was always a statement in Python to an expression – was defined and implemented much earlier than the structural pattern matching one.

3. X, Y, width, height

Blog update

Sun, 6 Aug 2023 00:00:00 +0200

OK, it’s been almost 5 years since something has been published here – it’s high time for an update.

First of all, it’s necessary to clarify that it’s not like there hasn’t been any writing in that time. I’ve exclusively (although unintentionally) been focusing on posting entries on Medium. You can view all of those on my Medium profile.

Since I’m slowly getting back into an increased blog publishing cadence^[1], I’m also ramping up the activity here, mostly with the following rule: this site is going to be the primary publishing target. Medium is going to receive the same blog posts (at least for now), but with a delay.

Speaking of which, a new blog post with some actual content :) will be up later today.

Cheers!

1. Hey, any positive number is greater than 0!

Travesty 0.9 now available!

Thu, 15 Mar 2018 00:00:00 +0100

(See the original blog entry to learn about Travesty in general)

First of all, supported Akka versions have been extended up to 2.5.11 .

Most importantly, Travesty can now generate diagrams from any graph, not only RunnableGraph / Graph[ClosedShape].

Here’s an example of a complex Flow shape being rendered:

As you can see, any open inlets and outlets will be marked automatically - and the relevant labels are configurable.

Hop onto the PROJECT PAGE for download and usage instructions!

Travesty 0.8 now available!

Sun, 11 Feb 2018 00:00:00 +0100

(See the previous blog entry to learn about Travesty in general)

The probably most important addition is preliminary typed diagram/graph support.

You need to add a bit of boilerplate to make it possible, but that just boils down to importing net.mikolak.travesty.registry._ and using either the .register or .↓ extension methods on your stages.

For example, this:

import net.mikolak.travesty
import net.mikolak.travesty.OutputFormat
import registry._ //adds special .↓ and .register methods to stages

val graph = Source.single("1").↓.via(Flow[String].map(_.toInt).↓).to(Sink.seq)

//render as image to file, PNG is supported as well
travesty.toFile(graph, OutputFormat.SVG)("/tmp/stream.svg")

Generates a diagram with appropriate element types:

Another major change is the ability to select which Graphviz engine is used. Specifically, if you have Graphviz installed on the target machine, you can add an application.conf with:

application.conf

travesty {
    engines = [CommandLine]
}

for a 10x+ speedup!

0.8 also features a fix for the Transform bug, and support for Akka 2.5.8, as well as 2.5.9 .

Hop onto the PROJECT PAGE for download and usage instructions!

Happy diagramming!

Travesty released!

Fri, 12 Jan 2018 00:00:00 +0100

What?

Travesty is a Scala library for doing two things with Akka Streams:

generating (untyped) structural diagrams of your Akka Streams, both as graphics and text (the latter useful for logging).
generating Tinkerpop 3 / Gremlin graph data structures - usable for e.g. writing tests that check stage sequencing in dynamically constructed Streams.

As you can imagine, I’ve originally developed this library with the growing stream structure of the "Akka Streams meets LibGDX" blogpost series. Here’s how the diagram for the current structure looks like:

Where to get it?

Head over to the GitHub repo, where you’ll find installation and usage instructions.

Reactive Streams in Scala meet a Game Engine - Part 7

Fri, 3 Nov 2017 00:00:00 +0100

Series' Table of Contents

In this episode…

We’re going to add input and interactivity to our game.

As a reminder…

We would like to have a game that:

requires relatively quick player reaction (so e.g. no turn-based games),
has a lot of events happening at any given moment,
and employs a large number of entities to do so.

Recall also one of our initial assumptions: mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions.

Getting input… in

LibGDX has two input modes - a polling one and an event-based one. The polling one can not only introduce inefficiencies, but also has the problem that:

Caution: If you rely on polling, you might miss events, e.g. a fast paced key down/key up. If you need to make sure a specific sequence of input action was completed, use event handling instead.

So let’s go with event handling.

In terms of supporting it, all we have to do is to create and register a listener. That’s simple enough, but there’s one more question - how do we transfer the events to the stream?

Well, we already have a solution - use a Source.actorRef[KeyboardInput], where KeyboardInput is our message type. So, all we need to do is to make a listener that "forwards" input events to a provided actor^[1].

Here’s the relevant code.

KeyboardProxy.scala

sealed trait KeyboardInput (1)
case class KeyUp(keycode: Int) extends KeyboardInput
case class KeyDown(keycode: Int) extends KeyboardInput
case class KeyTyped(character: Char) extends KeyboardInput

import akka.actor.ActorRef
import com.badlogic.gdx.InputAdapter

class KeyboardProxy(outActor: ActorRef) extends InputAdapter { (2)
  override def keyUp(keycode: Int) = {
    outActor ! KeyUp(keycode) (3)
    true
  }

  override def keyTyped(character: Char) = {
    outActor ! KeyTyped(character) (3)
    true
  }

  override def keyDown(keycode: Int) = {
    outActor ! KeyUp(keycode) (3)
    true
  }
}

1	Since LibGDX has no "input event data" types, we’ll make our own.
2	Taking in the `ActorRef` to forward to.
3	Forwarding key events to the actor.

Let’s go back to MainScreen and add a new Source for our events:

BulletHell.scala

case class GameState(delta: TickDelta, bodies: List[Body], events: List[KeyboardInput] = List.empty) (1)

1	`GameState` now supports listing current input events.

BulletHell.scala

class MainScreen extends ScreenAdapter {
....
  val inputSource = Source.actorRef[KeyboardInput](bufferSize = 0, OverflowStrategy.dropTail) (1)

1	`dropTail` since we want to react to latest input events in case of lag.

Let’s now modify our graph to integrate the source. Unfortunately there’s quite a few moving parts that can’t really be separated into shorter snippets - note that comments are out of sequence with the code, but instead roughly follow the design flow.

BulletHell.scala

    val inputLogger = (g: GameState) => { () => (7)
      {
        if (g.events.nonEmpty) {
          println(g.events)
        }
      }
    }

    val bufferSize = 100

    val graph =
      tickSource
        .zipWithMat(
          inputSource
            .batch(bufferSize, List(_))(_ :+ _) (1)
            .prepend(Source.single(List.empty[KeyboardInput]))  (3)
            .expand(elem =>
              Iterator.single(elem) ++ Iterator.continually(List.empty[KeyboardInput])) (2)
        )((gs, es) => gs.copy(events = es))(Keep.both) (4)
        .via(setUpLogic(List(generator, mover, worldUpdater, tickIncrementer, inputLogger)))
        .toMat(Sink.queue())(Keep.both)

    val ((sourceActor, inputActor), sinkQueue) = graph.run() (5)

    tickActor = Some(sourceActor)
    actionQueue = Some(sinkQueue)
    Gdx.input.setInputProcessor(new KeyboardProxy(inputActor)) (6)

1	`batch` does just that, it batches up elements when the downstream backpressures. In this case we’re simply batching by creating a `List`. Note that you will always get a `List[KeyboardInput]` down the line - so when there’s no backpressure, a list with a single element will be sent.
2	`expand` is there to do the opposite of `batch` - if downstream signals demand, `expand` will produce additional "synthetic" elements from the generated iterator. Its syntax is a bit weird: if you want to preserve the original element, it has to be produced as the first element of the iterator.
3	We need this because, currently, there’s no way of providing `expand` with a "seed" element in case when downstream signals demand at the very start^[2]. If we want the very first tick to go through, we need to use something like `prepend`. Note that this means the first tick will always be paired with an empty input list - but this is fine for our needs.
4	Here the argument part of the `zipMat` ends - we’re joining up the two sources by adding the list of events to the given `GameState`, and asking the materialization logic to keep both `ActorRefs`.
5	Why a nested tuple and not a triple? Because that’s how we ordered our materialization process - we could convert it into a triple, but there’s no point - since Scala’s pattern matching is smart enough to help us deconstruct the value anyway.
6	Here we register our `KeyboardProxy` listener as an LibGDX input handler. Note that it’s here that we use the `inputActor` that’s generated by the `Source[KeyboardInput]`.
7	A debug log for the events.

Interlude: A Failure to Communicate

The Problem

Now, you’ll occasionally notice the game will blackscreen and fail with the following message (or similar) :

[game-akka.actor.default-dispatcher-2]
 [akka://game/user/StreamSupervisor-0/flow-0-2-actorRefSource]
 Dropping element because there is no downstream demand: [GameState(2.14698E-4,List(),List())]

Since it’s from an actorRefSource and the type is GameState, we know this is from our tickSource. Let’s see where the message originates from ultimately. Here’s the offending code in Akka’s source:

akka/ActorRefSourceActor.scala

      if (totalDemand > 0L)
        onNext(elem)
      else if (bufferSize == 0)
        log.debug("Dropping element because there is no downstream demand: [{}]", elem)

as you can see, for this to trigger, the demand and buffer must be 0. Regarding the demand, going back to our render method:

BulletHell.scala

    tickActor.foreach { actor =>
      val bodyArray = ArrayGdx.of(classOf[Body])
      world.getBodies(bodyArray)
      actor ! GameState(delta, bodyArray.asScala.toList) (1)
    }
(2)
    import scala.concurrent.Await.result

    for {
      q <- actionQueue
      actions <- result(q.pull(), Duration.Inf) (3)
      a <- actions
    } {
      a()
    }

1	We’re sending our message here.
2	It arrives somewhere here. No demand was signalled to the stage at this time and the `TickDelta` element is dropped, and not passed to the subsequent stages.
3	We’re waiting infinitely here.

Finding the solution - source analysis

What signals demand? Multiple stages can do that - actionQueue.pull() is an obvious candidate, but if you muck around with the source code, you’ll find that ZipWith does so as well:

akka/ZipWith2

    override def preStart(): Unit = {
      pull(in0)
      pull(in1)
    }

In fact, if we create a simple example with a pulled-from Sink.queue(), or a zipWith:

object StreamPlay extends App {

  implicit val actorSystem = ActorSystem()
  implicit val mat = ActorMaterializer()

  val (tickSource, actionQueue) = Source.actorRef[String](0, OverflowStrategy.dropNew)
    //.zipWith(Source.single("BLAH"))((_,_)) (1)
    .toMat(Sink.queue())(Keep.both).run()

  //actionQueue.pull() (2)
  Thread.sleep(100)
  tickSource ! "BLAH"

}

1	Uncommenting this…
2	…or this will cause the message to not be dropped (at least on a reasonably fast machine).

and a sufficient timeout will make the stream work.

That answers the demand == 0 origin question. Of course, the bufferSize == 0 problem is our own damn fault:

BulletHell.scala

  val tickSource = Source.actorRef[GameState](bufferSize = 0, OverflowStrategy.dropNew)

Switching to bufferSize = 1 will make the problem go away. The relevant commit, with the fix and antecedent changes is here.

Finding the solution - debugging

The buffet

Of course, taking a deductive stroll through the source code is refreshing every now and again, but quickly becomes tedious in day-to-day work. It would be good, then, to have a tool that helps us with that.

Unfortunately, at least as open source tools go, there’s not much to use. Sure, there are a few, but they’re hopelessly outdated.

Now, falling back to plain Akka, we have options such as Kamon and akka-visualmailbox. Due to relative ease of setup, we’ll go with the latter.

So, we only need to add:

application.conf

akka.actor.default-mailbox.mailbox-type = "de.aktey.akka.visualmailbox.VisualMailboxType"

run the visualization container image:

console

docker run -ti --rm -p 8080:8080 -p 60009:60009/udp ouven/akka-visual-mailbox-visualization

and open the browser on http://localhost:8080.

However, once we run the (non-buggy iteration of) program, we get something like:

Where are the other stages?

Getting to the point

Currently, Akka Streams employs "auto-fusing" per default. In simple terms, these means that stages that would be executed sequentially are "joined" (or, indeed, "fused") into a single actor that does the same work. More details can be found here.

The simple workaround is to insert async boundaries into our stream, "forcing" the materializer to instantiate separate actors. And this is how it looks like:

BulletHell.scala

      tickSource.async (1)
        .zipWithMat(
          inputSource.async (1)
            .batch(bufferSize, List(_))(_ :+ _)
            .prepend(Source.single(List.empty[KeyboardInput]))
            .expand(elem =>
              Iterator.single(elem) ++ Iterator.continually(List.empty[KeyboardInput]))
        )((gs, es) => gs.copy(events = es))(Keep.both)
          .async (1)
        .via(setUpLogic(List(generator, mover, worldUpdater, tickIncrementer, inputLogger)))
          .async.toMat(Sink.queue())(Keep.both) (1)

1	Manually inserted boundaries, stopping the materializer from fusing the stages.

And now we finally can see the "normal" state:

as well as the failure condition:

Alternatives?

The only viable alternative is Kamon. My colleague has provided an overview of that lib already, so - given that Kamon also only provides a "basic-Akka-level" view of things - I’ll defer those interested to that.

OK - what the hell?

It seems like we have two more problems now:

The support for sensible (and open-source) Akka Streams debugging is poor.
We got hit with an unexpected problem completely out of the blue, with seemingly no warning from the library.

The 1st issue unfortunately remains a fact of life for the time being. One probable reason is the internal complexity of Akka Streams - due to the materialization mechanism there are additional layers of abstraction to consider and interpret.

As already said, the blame for the 2nd problem lies squarely on us - not only the error condition is dully logged by Akka, we should have seen it coming when we used an OverflowStrategy with drop in the name.

The takeway here is:

use DEBUG mode for local development (applies for most dev work really, regardless of particular libraries),
avoid defining empty buffers, especially for "externally communicating" stages (such as Source.actorRef), unless you know what you’re doing.

Making the input count

Let’s add some actual interactivity into the mix. Keeping it simple, we’ll just add an additional, differently-sized sphere that is controllable via key presses.

The expedient solution is to modify our createSphere method, adding two arguments:

one that determines the size of the sphere,
another that makes the "player" and "enemy" objects distinct somehow.

The second one is a bit non trivial. We could pop yet another piece of data into our GameState, but there’s a quick alternative - each Body contains a userData attribute that is specifically intended to attach custom information to the entities, so let’s use that.

First, we need a coproduct/enum for our representation:

BodyType.scala

sealed trait BodyType
case object Player extends BodyType
case object Enemy extends BodyType

Then, we modify our createSphere accordingly:

BulletHell.scala

  private def createSphere(center: Vector2, bodyType: BodyType, radius: Float = 1) = { (1)
    val bodyDef = new BodyDef
    bodyDef.`type` = BodyType.DynamicBody
    bodyDef.position.set(center)

    val circle = new CircleShape()
    circle.setRadius(radius) (1)

    val fixtureDef = new FixtureDef()
    fixtureDef.shape = circle
    fixtureDef.density = 1f

    val body = world.createBody(bodyDef)
    body.createFixture(fixtureDef)
    fixtureDef.shape.dispose()
    body.setUserData(bodyType) (1)
  }

1	Modified lines.

Next, we augment our generator function:

    val generator = (g: GameState) => { () =>
      {
        if (g.bodies.isEmpty) {
          for (_ <- 1 to config.world.gen.NumCircles) {
            val randomLocation = new Vector2(Random.nextInt(config.world.Width.size),
                                             Random.nextInt(config.world.Height.size))
            createSphere(randomLocation, Enemy) (1)
          }

          //player body
          createSphere(new Vector2(config.world.Width.size, config.world.Height.size).scl(0.5f),
                       Player,
                       radius = 2f) (2)
        }
      }
    }

1	Adding the marker type here…
2	…and the player-controlled object here, twice as large and placed in the middle of the play area initially.

As well as the mover function:

...
          for (body <- g.bodies if body.getUserData == Enemy) { (1)
...

1	We now only want the non-player entities to be self-propelled, of course.

Finally, we add a handler function for the input:

    val inputHandler = (g: GameState) => { () =>
      {
        for {
          playerBody <- g.bodies.find(_.getUserData == Player)
          e <- g.events
        } {
          val inputMults = e match {
            case KeyUp(keycode) => (1)
              keycode match {
                case Keys.LEFT  => (-1f, 0f) (2)
                case Keys.RIGHT => (1f, 0f)
                case Keys.UP    => (0f, 1f)
                case Keys.DOWN  => (0f, -1f)
                case _          => (0f, 0f)
              }
            case _ => (0f, 0f)
          }

          val v = (new Vector2(_: Float, _: Float)).tupled(inputMults).scl(1000f) (3)
          playerBody.applyForceToCenter(v, true) (4)
        }
      }
    }

...
        .via(setUpLogic(List(generator, mover, worldUpdater, tickIncrementer, inputHandler))) (5)
...

1	We’re interested only in `KeyUp` events (arbitrary, it could have also been `KeyDown`, as long as we pick one).
2	The events are encoded into a tuple that represents the (X,Y) directional displacement of our sphere.
3	The tuples are converted into a sufficiently large force vector…
4	…which is applied to the controllable sphere.
5	Let’s not forget to add this function to our logic stream (we also removed the logger).

As you can see, this works without any problems:

Your browser does not support the video tag.

Here’s the relevant commit.

Discussion

At this time it might be interesting to see how other "reactive" approaches handle time-sensitive input.

During the relevant talk Aleksandar Prokopec describes the handling of input via a combinations of:

Reactives - congruous to Rx’s Observables and Stream’s stages, and
Signals - which seem to correspond to Rx’s BehaviorSubject or Stream’s expand Flow stage ^[3].

rxjava-libgdx just uses a system of Observables which are based on the "reification-of-input" approach that we also used here (i.e. converting listener method calls into discrete data objects).

What’s next?

As a reminder, we’ve mentioned the following problems in the preceding part:

the worldUpdater relies on closing over the World instead of being supplied with it;

similarly, the tickIncrementer not only reads but also changes an "external" value;

the rendering logic is outside the system, because it needs to be invoked in a specific sequence in relation to the other operations, and we have no control over that sequence currently;

related to all of the above, we have no way of signalling the stream setup logic what information we have in our executor functions;

due to the lack of a definable ordering, we also run the risk of race conditions, despite everything being "executed" in the same thread - e.g. if we remove a body in one place, it can still be referenced in another, resulting in a crash of the app;

we’re starting to have a god object, it would be good to remove the entire "stream engine" portion out of MainScreen soon.

If, right now, you’re staring dumbfounded at this list, wondering how we managed to solve any of these problems - then you’re on the right track!

The truth is, keyboard input integration has taken too much effort to tackle the pertinent problems. However, in the immortal words of Bob Ross, this is a "happy accident". Why?

Well, let’s see:

we now see that issue 4. will come to bite us in the hindquarters if we don’t do anything about it - already we needed to smuggle metadata in LibGDX’s objects, and this was only a trivial marker - what about potentially needed stuff such as health points, movement energy, etc. etc.?
the same applies to point 5.: we now have a specific, immediate problem to tackle - what will happen once we add collision detection and health point depletion? In our current state - chaos!

Furthermore (and related to point 5.), if you compare our current solution to other "reactive" implementation, you’ll find that it’s quite inflexible - in particular, it doesn’t support any sort of combinators (.map, .flatMap, .filter, etc. etc.). Taking together that conundrum with the previously described issue, the following are all examples of "component" types we want to support:

(List[Body]) => () => Unit (1)
(List[Body]) => SomeData (2)
(SomeData) => () => Unit (3)
Flow[(SomeData, KeybardInput), () => Unit, _] (4)
Flow[(List[Body], TickDelta), SomeData, _] (4)

1	Sometimes we wont need the tick info…
2	…sometimes we’d want to create custom data…
3	…sometimes all we need is that custom data…
4	…and sometimes we’d wish to use the power of stage combinators.

At the end of the day, we have a more comprehensive view of what we need to clean up, and more importantly, how. And that’s what the next part will (now, definitely, I promise!) cover.

That’s not to say we haven’t accomplished anything - we now enjoy at least a minimal semblance of interactivity, and, so far, the code runs relatively smoothly. With that in mind, see you again soon!

1. A fancy name for this kind of logic is "reification", i.e. "making a real thing out of something" - in this case, making discrete objects out of method calls.

2. although a PR for that is in the works.

3. It’s difficult to say where the analogies lie in the actually-released library, i.e. Reactors, as the terminology is different and apparently non-isomorphic to the one in the talk

Reactive Streams in Scala meet a Game Engine - Part 6

Sun, 3 Sep 2017 00:00:00 +0200

Series' Table of Contents

In this episode…

We’re going to make the first step in reorganizing our stream to be more modular and thread-safe.

As a reminder…

We would like to have a game that:

requires relatively quick player reaction (so e.g. no turn-based games),
has a lot of events happening at any given moment,
and employs a large number of entities to do so.

Recall also one of our initial assumptions: mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions.

Modularizing the game’s logic

Currently the vast majority of everything that "happens" in our game is located in one giant block of code. This is unsustainable, and not what we want in the long run.

So, let’s split up the singular flow stage. Ideally, it’s broadly two-phased:

One where everything except thread-sensitive stuff happens;
the other where all actions bound to the UI thread happen.

The simplest way to implement the structure defined above is to generate functions in the first phase, and then invoke them in the second.

Figure 1. Sketch of our planned approach.

Code-wise, we should start with defining an alias, for convenience’s sake :

BulletHell.scala

class MainScreen extends ScreenAdapter {
   type Action = () => Unit
...

Now, let’s code in the "engine" of the first phase:

BulletHell.scala

class MainScreen extends ScreenAdapter {
...
  private def setUpLogic(
      elements: List[(GameState) => Action]): (1)
        Flow[GameState, Seq[Action], NotUsed] =
    Flow.fromGraph(GraphDSL.create() { implicit b =>
      import GraphDSL.Implicits._ (3)
      val scatter = b.add(Broadcast[GameState](elements.size)) (4)
      val gather = b.add(ZipN[Action](elements.size)) (5)

      for (e <- elements) {
        scatter ~> b.add(Flow.fromFunction(e)) ~> gather (6)
      }

      FlowShape(scatter.in, gather.out) (7)
    })
...

1	We’re accepting a list of functions that create actions based on our current `GameState`
2	…and returning a `Flow` that aggregates the resulting actions into one sequence per step.
3	Akka Streams' Graph DSL boilerplate.
4	The `Broadcast` stage simply provides `n` outlets with every input element.
5	The `ZipN` stage waits for every one of its `n` inlets to produce an element, and aggregates them into a sequence.
6	Wiring everything together…
7	…and producing a "black box" blueprint with all the logic hidden behind the `FlowShape`.

Now let’s rewrite our logic into a series of modules and replace our original tickFlow with the new logic flow:

BulletHell.scala

  override def show() = {
  ...
    val generator = (g: GameState) => { () => (1)
      {
        if (g.bodies.isEmpty) {
          for (_ <- 1 to config.world.gen.NumCircles) {
            val randomLocation = new Vector2(Random.nextInt(config.world.Width.size),
                                             Random.nextInt(config.world.Height.size))
            createSphere(randomLocation)
          }
        }
      }
    }

    val mover = (g: GameState) => {
      import config.world.gen
      def randomForceComponent = Random.nextInt(2 * gen.MaxForce) - gen.MaxForce

      () =>
        {
          for (body <- g.bodies) {
            if (tick % gen.ForceApplyTickInterval == 0) {
              body.applyForceToCenter(randomForceComponent, randomForceComponent, true)
            }
          }
        }
    }

    val tickIncrementer = (_: GameState) => { () =>
      {
        tick += 1
      }
    }

    val worldUpdater = (g: GameState) => { () =>
      {
        world.step(g.delta, 6, 2)
      }
    }

    val graph = tickSource
      .via(setUpLogic(List(generator, mover, worldUpdater, tickIncrementer)))
      .to(Sink.ignore)
  ...

1	Every component now returns a function that can be called later.

There’s still one loose end, namely actually invoking the generated actions.

Introducing more thread safety

There are two ways to do that.

Using materialization

We’ve already taken into advantage of materialization to obtain an input to our stream, by way of a materialized ActorRef.

However, materialization can provide an arbitrary number of values, as illustrated by this diagram from the docs^[1]:

We turn to the nice list of available stages, again, from the docs^[2]. Happily, we find there’s a couple of sink stages that produce "dynamic" output, e.g. actorRef which is a dual to Source.actorRef.

However, the most convenient method for us here is Sink.queue. Materializing it generates a SinkQueue that can be polled for every separate element that arrives at the sink.

Let’s add a storage var for this queue to our MainScreen, as well as a type alias for abstracting away implementation details:

BulletHell.scala

class MainScreen extends ScreenAdapter {

  ...
  type ActionQueue = SinkQueueWithCancel[Seq[Action]]

  ...
  var actionQueue: Option[ActionQueue] = None

Now all that’s left is to replace our Sink.ignore with the queue…

BulletHell.scala

  override def show() = {
  ...

    val graph = tickSource
      .via(setUpLogic(List(generator, mover, worldUpdater, tickIncrementer)))
      .toMat(Sink.queue())(Keep.both) (1)

    val (sourceActor, sinkQueue) = graph.run()

    tickActor = Some(sourceActor)
    actionQueue = Some(sinkQueue)
  ...

1	`toMat` hints that the materialized value from the sink should be preserved. `Keep.both` is a second argument of `toMat`, instructing that the "original"/"left" value should be combined with the "other"/"right" value to a tuple.

…and invoke the resultant actions:

BulletHell.scala

  override def render(delta: TickDelta) = {
    ...

    import scala.concurrent.Await.result

    for {
      q <- actionQueue (1)
      actions <- result(q.pull(), Duration.Inf) (2)
      a <- actions
    } {
      a() (3)
    }
  ...

1	Since `actionQueue` is an `Option`.
2	The `pull()` method returns a value of `Future[Option[Seq[Action]]]`, so we need to wait for the result and deconstruct the `Option`.
3	Each action gets invoked here.

The code is available under this commit. If you check out and execute this ref, you’ll find our game runs exactly the same, meaning we accomplished our goal.

Assigning a custom dispatcher

(This part is optional to the understanding of the text as a whole.)

On first glance, it seems it would be possible to instead define a Sink.foreach stage with a dispatcher that runs on the game’s UI thread.

Furthermore, it sounds like the Testkits’s CallingThreadDispatcher would be the way to go. Let’s try that out

BulletHell.scala

    val graph = tickSource
      .via(setUpLogic(List(generator, mover, worldUpdater, tickIncrementer))) (1)
      .to(Sink.foreach[Seq[Action]](actions => {
        actions.foreach(_()) (2)
        println(Thread.currentThread().getName) (3)
      }).withAttributes(Attributes(Dispatcher(CallingThreadDispatcher.Id)))) (4)


    tickActor = Some(graph.run()) (5)

1	Same code as in the `SinkQueue` variant.
2	Iterating and calling all actions.
3	Debug statement to see whether we’re on the current thread.
4	Dispatcher’s in Akka are created dynamically from the configuration and referenced by IDs.
5	Going back to the same code as in the previous part of this blog.

However, when we run it, we would see e.g.:

game-akka.actor.default-dispatcher-4
game-akka.actor.default-dispatcher-3
game-akka.actor.default-dispatcher-4
game-akka.actor.default-dispatcher-3
game-akka.actor.default-dispatcher-4

whereas the UI thread is named LWJGL Application. If we think for a minute, that makes sense - the calling thread here is a thread from the pool used by the previous stages (the ones generating the actions, or rather the ZipN substage).

So, what we would need to do is to setup a custom dispatcher that would enqueue Runnable s ^[3], and expose a method named e.g. popAndRun() that would take the first Runnable out of the queue and execute it in the current thread. This popAndRun() would then be invoked during MainScreen#render().

However, since dispatchers cannot be created programmatically, we would need to employ some trickery in order to invoke such a method. I think it would work, specifically through the Dispatcher-lookup system and several casts, but would be a hacky way to implement what we already have achieved anyway.

The only advantage of that would be that we stay within the Akka Streams API directly (as opposed to the queue materialization approach). However, I don’t think this alone merits further investigation into a kludgy alternative implementation.

Loose ends

Whatever the approach, we can see that the current architecture isn’t without problems:

the worldUpdater relies on closing over the World instead of being supplied with it;
similarly, the tickIncrementer not only reads but also changes an "external" value;
the rendering logic is outside the system, because it needs to be invoked in a specific sequence in relation to the other operations, and we have no control over that sequence currently;
related to all of the above, we have no way of signalling the stream setup logic what information we have in our executor functions;
due to the lack of a definable ordering, we also run the risk of race conditions, despite everything being "executed" in the same thread - e.g. if we remove a body in one place, it can still be referenced in another, resulting in a crash of the app;
we’re starting to have a god object, it would be good to remove the entire "stream engine" portion out of MainScreen soon.

In the next part, we will be solving these problems, and also adding some input to the game, finally making interactive. To that end, we’ll be using architecture templates that app developers find familiar (as an exercise, you may try to figure out what they will be given the current state of our codebase and the enumerated problems).

1. On the other hand, if you find yourself using materialized values for anything other than inputs and outputs, consider that a code smell.

2. Although it’s sadly missing links to the actual implementations.

3. To be specific, the ExecutorServiceFactory generated by its ExecutorServiceFactoryProvider would do that.

Reactive Streams in Scala meet a Game Engine - Part 5

Sun, 20 Aug 2017 00:00:00 +0200

Series' Table of Contents

In this episode…

We’re finally making our game do something.

As a reminder…

We would like to have a game that:

requires relatively quick player reaction (so e.g. no turn-based games),
has a lot of events happening at any given moment,
and employs a large number of entities to do so.

Recall also one of our initial assumptions: mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions.

A bit of refactoring

Magicks!

We’re now well into our development of our little game, which merits a critical look back into what we’ve coded in so far.

Principally, we seem to have a bit of magical numerology in our dimensional definition, specifically in both our code and the desktop version:

core/BullletHell.scala

camera.setToOrtho(false, 800, 480)

desktop/Main.scala

cfg.height = 480
cfg.width = 800

They’re both in sore need to be moved to a globally settable constant value. However, they both seem to be aligned in form and function, so let’s create a value class that can hold them:

case class Dim(d: Float) extends AnyVal { (1)
  def size = d.toInt
}

1	Defining a value class.

I confess here on cheating a little bit - I know this the, erm, value of this value class is going to shine through in a moment. However, I’d like to point out that nevertheless value classes are cheap not only in terms of writing them, but also in terms of executing the relevant code. Because all this nets executable code within static invocations, as long as user-defined value classes can be used, they remain useful entities to base our domain model upon on.

Come worst case, time spent writing them will be undone by (sometimes literally) a couple of editor actions.

Reflowing!

That still leaves us the problem of centralizing our generalized code. We could make a proper config, but a quick compromise trick, useful for sketching out configs, is to store everything in a nested object. In our case, this means:

object config {
  object world {
    val Width = Dim(160)
    val Height = Dim(100)
  }
}

The pattern has several advantages:

requires no additional dependencies,
it provides practically all of the benefits of a "proper" config during the stage of development where code modifications are abundant,
since config keys are actually Scala stable identifiers, code assist/renaming etc. works out of the box,
once the point of diminishing returns vs a "proper" config is reached, a swap to it can be effected via a relatively simple regex.

So, our calling code is now:

core/BullletHell.scala

camera.setToOrtho(false, config.world.width.size, config.world.height.size)

We’re also going to decouple the size of the desktop window and increase it a bit:

desktop/Main.scala

cfg.height = 640
cfg.width = 1024

Note that 1024/640 = 160/100 = 1.6.

OK, but why 160 x 100?

Adding Box2D basics

Well, our next change will be introducing physical objects into our game, and, for that, we’re be going with a physics module that integrates Box2D with LibGDX.

Box2D is a widely-used C++ framework for simulating physics in 2 dimensions (as the name suggests). It is both simple and fairly powerful. However, as the FAQ outlines, non-stationary objects should be no larger than around 10 units/meters^[1].

So, the size of 160 x 100 is sufficient to have interactable, visible objects in the sweet spot of Box2D preferred sizes, and conforms to the aspect ration of our viewable window.

Having said that, let’s now add the following:

core/BullletHell.scala

val world = new World(new Vector2(0, 0), false) (1)

val debugRenderer = new Box2DDebugRenderer() (2)

1	This is the core Box2D management object. The first argument describes the gravity vector (we want to have a weightless environment, so it’s `(0,0)`), the second is a flag to enable an optimization technique.
2	A very useful class when prototyping, it draws wireframes of all physics objects for you.

Before we modify our Flow, we need to include enhance the tick element to account for our extended game state.

Specifically, we need to include at minimum all Box2D bodies currently present. Bodies represent physical objects in Box2D.

So, instead of a simple TickDelta, we’ll now be producing this:

case class GameState(delta: TickDelta, bodies: List[Body])

Let’s change the render function to account for that:

core/BullletHell.scala

override def render(delta: TickDelta) = {

    tickActor.foreach { actor =>
      val bodyArray = ArrayGdx.of(classOf[Body])
      world.getBodies(bodyArray) (1)
      actor ! GameState(delta, bodyArray.asScala.toList)
    }

    ...

1	Song-and-dance necessary to get all current objects - this pass-by-reference approach is both due to Box2D C++ roots and for efficiency’s sake.

Now we modify our flow. For starters, let’s create a couple of circular bodies in random places. Here’s how a utility method for creating a new circle body looks like:

core/BullletHell.scala

  private def createSphere(center: Vector2) = {
    val bodyDef = new BodyDef (1)
    bodyDef.`type` = BodyType.DynamicBody (2) (3)
    bodyDef.position.set(center) (4)

    val circle = new CircleShape()
    circle.setRadius(1)

    val fixtureDef = new FixtureDef() (5)
    fixtureDef.shape = circle
    fixtureDef.density = 1f

    val body = world.createBody(bodyDef)
    body.createFixture(fixtureDef)
    fixtureDef.shape.dispose() (6)
  }

1	A body definition is simply an abstract blueprint for a body.
2	Since `type` is a reserved word in Scala, we have to use the backtick notation here.
3	Dynamic bodies are ones that behave exactly how you would expect a physical body to - they are subject to gravity, collisions with other objects, user-defined forces, etc.
4	The `Vector2` argument is used as the `(x,y)` coordinates of the center of our circle.
5	Bodies are composed of one or more fixtures. In our case we only have one, with a circular shape and a density of 1 kg/m³ (so e.g. like mainstream American beer).
6	More game-engine-specific song-and-dance. If we’d generate a considerable number of these shapes, we’d cache this object, but here it’s unnecessary.

Let’s add a couple of new config values:

object config {
  object world {
    object gen {
      val NumCircles = 3
      val MaxForce = 300
      val ForceApplyTickInterval = 100
    }
    ...

Now we’re ready to modify our flow. We need to match the new game state object, add some circles in random positions, and make them move, all in our show method:

core/BullletHell.scala

val tickSettingFlow = {

  Flow[GameState].map { case gs@GameState(delta, bodies) => (1)
    if(bodies.isEmpty) { (2)
      for(_ <- 1 to config.world.gen.NumCircles) {
        val randomLocation = new Vector2(
                                Random.nextInt(config.world.Width.size),
                                Random.nextInt(config.world.Height.size)) (3)
        createSphere(randomLocation)
      }
    } else { (4)
      import config.world.gen (5)
      def randomForceComponent = Random.nextInt(2*gen.MaxForce)-gen.MaxForce

      for(body <- bodies) {
        if(tick % gen.ForceApplyTickInterval == 0) { (6)
          body.applyForceToCenter(randomForceComponent, randomForceComponent, true)
        }
      }
    }

    world.step(delta, 6, 2) (7)
    tick += 1
    gs
  }
}

1	Matching to the new game state.
2	Starting situation, lets create our bodies.
3	Completely random location within our world.
4	Our game logic "proper".
5	As mentioned before, the "quick config" scheme gives us the advantage of value keys being stable identifiers, this is an example of what this approach provides.
6	We randomly apply a force every X ticks.
7	The `World` needs to be manually "stepped over". The second and third parameters are the recommended values for the engine’s constraint solver.

We also have to add this:

debugRenderer.render(world, camera.combined)

to the end of our show method to fire up the aforementioned renderer.

And this is basically it! Here’s how a possible run looks like:

Your browser does not support the video tag.

Exception reporting

As a final touch for this episode, let’s add a decider for our stream. We need it, because Akka Streams has one notorious design quirk - exceptions are silently swallowed by default ^[2].

The normal way to go around that is outlined in this SO thread.

In our case this would mean:

core/BullletHell.scala

val loggingDecider: Supervision.Decider = { e =>
    println(s"Exception when processing game loop: $e")
    Supervision.Stop
}
implicit val actorSystem = ActorSystem("game")
implicit val materializer = ActorMaterializer(ActorMaterializerSettings(actorSystem)
    .withSupervisionStrategy(loggingDecider))

Now we’re sure that if an exception occurs in our flow, we’ll now about it.

The code for the current part is available here.

Next up

Our flow works, but is still not thread-safe, and essentially one big logic blob. We’re going to improve that.

1. Box2D uses meters as size units.

2. That’s not strictly true - there are incidental logs that show up on the DEBUG level, indicating that something happened, but that still does not list the exact cause of why the stream failed.

New Layout

Tue, 1 Aug 2017 00:00:00 +0200

As you may have noticed there has been a layout refresh. The old look was never meant to be permanent (and, regrettably, last as long as it did), so what you’re seeing now is somewhat closer to what I intended to be hear.

Anyway, Part 5 of the Akka Streams/LibGDX series is coming soon. In the meantime, enjoy the existing content!

Reactive Streams in Scala meet a Game Engine - Part 4

Fri, 7 Jul 2017 00:00:00 +0200

Series' Table of Contents

In this episode…

…we’re finally ready to add Akka Streams code to our setup, by implementing the heart of our game logic stream - the tick Source.

As a reminder…

We would like to have a game that:

requires relatively quick player reaction (so e.g. no turn-based games),
has a lot of events happening at any given moment,
and employs a large number of entities to do so.

Recall also one of our initial assumptions: mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions.

Scouting out the solution space

Let’s define our core problem first - we need to somehow transfer the information about each incoming tick into our stream. The first order of business is to find a way to inject that information into the stream.

It’s probably tempting to create a custom publisher-listener solution for that, but we would wind up reinventing the wheel this way. Instead, let’s look whether we can find out how the Akka Streams' API deals with the problem.

As a matter of fact, there is a Source.tick factory method that handles a similar scenario. However, the ticks here are generated internally, so it’s of no use for us.

However, we see that two other useful factory methods exist:

Both provide us with an ActorRef that we can use to pump ticks into our stream.

Recall: in Akka Streams, running a graph can return a non-unit value - this is called the materialized value of a graph ^[1].

Usually, the materialized value is used to get some "result" out from running the graph. However, what’s important to remember is that any graph stage can generate a materialized value, not only a Sink.

In fact, it is almost equally frequent to use a materialized value not only to get some output out of the graph, but also provide input into it ^[2]. This is what we are doing here - once the graph is ran, the ActorRef we get as the return result serves as our "messenger" between our libgdx code and the Akka Streams graph.

We’re going to investigate both to see whether they help us solve our problem, but first, let’s…

Setup

First of all, that Float for the tick delta time looks ungainly, so we’ll introduce a package object to deal with that:

stream_bullethell/package.scala

package net.mikolak

package object stream_bullethell {
  type TickDelta = Float
}

Next, let’s create a framework for our stream - we’re going to be printing ticks from our stream now:

class MainScreen extends ScreenAdapter {

  lazy val camera = new OrthographicCamera()
  val batch: SpriteBatch = new SpriteBatch()

  var tick = 1L

  implicit val actorSystem = ActorSystem("game")
  implicit val materializer = ActorMaterializer() (1)

  val tickSource: Source[Nothing, ActorRef] = ??? (3)
  var tickActor: Option[ActorRef] = None (4)

  lazy val font = {
    val f = new BitmapFont()
    f.getData.setScale(2f)
    f
  }

  override def show() = {
    camera.setToOrtho(false, 800, 480)

    val tickSettingFlow = Flow[TickDelta].map { td =>
      tick += 1
      td
    } (5)
    val graph = tickSource.via(tickSettingFlow).to(Sink.ignore) (6)

    tickActor = Some(graph.run()) (7)
  }

  override def render(delta: TickDelta) = {
    tickActor.foreach(_ ! delta) (8)

    //print tick
    Gdx.gl.glClearColor(0, 0, 0.5f, 1)
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT)
    camera.update()
    batch.setProjectionMatrix(camera.combined)
    batch.begin()
    font.draw(batch, s"Tick: $tick", 0, font.getCapHeight)
    batch.end()
  }

  override def dispose(): Unit = {
    actorSystem.terminate() (2)
  }
}

1	Standard initialization song-and-dance.
2	Deinit here ^[3].
3	Placeholder for our `Source`, note the materialized value type (`ActorRef`).
4	Placeholder for the materialized value.
5	Defining a simple `Flow` that increments the `tick` value from the Stream level, and passing the `TimeDelta` value through.
6	Connecting all parts…
7	…and materializing the graph with `run`.
8	Here’s where we communicate the occurrence of a tick with the stream.

The concept is straightforward: we’re using a materialized value (an ActorRef) to ping the source with a tick delta time every time a tick occurs.

You can play around with the source code for this stage here.

`Source.actorPublisher`

To use this method, we need to implement an actor with an ActorPublisher trait.

Basically, we need to create a simple queue implementation that pushes new items using onNext, whenever there is demand downstream. Here’s how the implementation might look like:

class GameTickPublisherActor() extends Actor with ActorPublisher[TickDelta] {

  private var updateStack = List.empty[TickDelta]

  override def receive = {
    case delta: TickDelta =>
      updateStack :+= delta (1)

      if (isActive && totalDemand > 0) { (2)
        val (toTransmit, toPreserve) =
          updateStack.splitAt((updateStack.length - totalDemand.toInt).max(0))
        toTransmit.foreach(onNext) (3)
        updateStack = toPreserve
      }
  }
}

1	Adding to our element queue.
2	Checking whether we should push something downstream.
3	Transmitting `totalDemand` elements to the upstream.

(Note that onNext, isActive and totalDemand are all references introduced by the ActorPublisher trait)

Then, all we need to do is fill out our Source placeholder in MainScreen:

  val tickSource: Source[Nothing, ActorRef] = Source.actorPublisher(Props[GameTickPublisherActor])

Here’s the source code for this implementation.

And we’re getting our ticks:

Your browser does not support the video tag.

At first glance, it looks like it’s working, although not completely smoothly. Let’s try out the other approach.

`Source.actorRef`

This one’s even simpler, all we have to do is set up the source:

   val tickSource = Source.actorRef[TickDelta](0, OverflowStrategy.dropNew)

actorRef requires two parameters:

bufferSize - which we set to 0 to disable the buffer,
overflowStrategy - does not actually matter, since buffer is disabled - although dropNew is basically the implementation’s behavior in this case.

Here’s how our tick demonstrator runs now:

Your browser does not support the video tag.

And here’s the source code.

We can now plainly see that the actorPublisher version fails to produce values expediently, compared to the variant currently discussed. Why is this the case?

Keeping things simple

Obviously there’s a problem somewhere with the first implementation, but where? Let’s start with logging toTransmit on every tick. It turns out we will get something like:

List()
List()
List()
List()
List()
List(0.01748508)
List(0.016564002, 0.015894385)
List(0.017008292, 0.017062303, 0.015971143)
List(0.01700024, 0.01696568, 0.016022582, 0.01697306)

This is obviously incorrect, since we should always check for a positive demand (and therefore should be sending non-empty lists).

Of course, this implies we have the third member of the unholy duo of programming mistakes, namely an off-by-one error. Specifically, here:

          updateStack.splitAt((updateStack.length - totalDemand.toInt).max(0))

we max to 0 instead of 1. Correcting the line to max(1) will make both implementations run smoothly.

Moreover, seeing as how the actorPublisher implementation has no advantage over the actorRef one, and is substantially more convoluted, we’ll be sticking with the latter.

Up next

We now have a tick source, so we will be able to liven our game’s world a little by adding some entities mulling about.

Also, we have completely neglected thread safety up until now - we’ve been updating the game state (in this case, the tick var) from at least two different threads. We’ll be working towards eliminating this kludge as well.

Stay tuned!

1. Confusingly, running a graph is also called "materializing" it…

2. You can do either, or both at the same time - consult the documentation for details.

3. This is actually incorrect, but the error is not critical for our purposes.

Reactive Streams in Scala meet a Game Engine - Part 3

Wed, 5 Jul 2017 00:00:00 +0200

Series' Table of Contents

In this episode…

…we will start the actual development of the game by planting some boilerplate, taking a crash course in libgdx, and setting the stage for the actual streaming implementation.

Libgdx Basics

Project Structure

Libgdx, as mentioned previously, is a popular JVM framework for game design. Rudimentary Scala support exists - in fact, our game skeleton is based on ajhager’s g8 template, or rather, its fork by Darkyenus.

Because libgdx is a multi-platform engine, projects using it are always multi-module, split into:

a core module that contains the generic base of the game,
one or modules, each for a different target’s specific code, e.g. desktop, android, ios, etc.

In our case we will be only implementing a desktop version, so we will have two modules: core, and desktop.

Boilerplate classes

The core 's structures starts with the primary entrypoint class - a subclass of Game. Each platform variant creates an instance of this class during initialization.

In turn a Game operates on a number of Screen s, exactly one of each is always active. The Screen subclasses usually contain most of the game’s logic, the Game subclass should only provide minimal global controller "glue".

Both Game and Screen are coded in by implementing the provided lifecycle methods, create, pause, resume, dispose etc. ^[1].

With the general introduction out of the way, let’s see how a minimal game looks like. Here’s the core logic:

package net.mikolak.stream_bullethell

import com.badlogic.gdx.graphics.GL20
import com.badlogic.gdx.{Game, Gdx, ScreenAdapter}

class BulletHell extends Game {

  override def create(): Unit = { (1)
    setScreen(new MainScreen) (2)
  }

}

class MainScreen extends ScreenAdapter { (3)

  override def render(delta: Float) = {
    Gdx.gl.glClearColor(0, 0, 0.5f, 1) (4)
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT)
  }

}

1	Overriding a lifecycle method here.
2	Setting the active screen to our implementation.
3	Using an abstract Adapter class instead of the actual `Screen` interface for convenience.
4	Painting the background color to dark blue.

We then define a platform specific implementation, in this case, a "desktop"^[2] JRE target:

package net.mikolak.stream_bullethell

import com.badlogic.gdx.backends.lwjgl.{LwjglApplication, LwjglApplicationConfiguration} (1)

object Main extends App {
    val cfg = new LwjglApplicationConfiguration
    cfg.title = "stream-bullethell-demo"
    cfg.height = 480
    cfg.width = 800
    cfg.forceExit = false
    new LwjglApplication(new BulletHell, cfg) (2)
}

1	"Desktop"-specific imports.
2	Pointing to our `Game` subclass.

This leaves us with an 800px by 480px screen with a dark blue background.

Erecting a base camp

The previous listing doesn’t leave us much room for experiment, so let’s spice things up a little by adding a proper Hello World message to MainScreen:

class MainScreen extends ScreenAdapter {

  private lazy val camera = new OrthographicCamera() (1)
  private val batch: SpriteBatch = new SpriteBatch() (2)

  private lazy val font = {
    val f = new BitmapFont()
    f.getData.setScale(2f)
    f
  } (3)

  override def show() = {
    camera.setToOrtho(false, 800, 480) (4)
  }

  override def render(delta: Float) = {
    Gdx.gl.glClearColor(0, 0, 0.5f, 1)
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT)
    camera.update()
    batch.setProjectionMatrix(camera.combined) (5)
    batch.begin() (6)
    font.draw(batch, "Hello libgdx!", 0, font.getCapHeight) (7)
    batch.end()
  }

}

1	The camera defines how the player "looks" at the object on the scene. In this case, we are using an Orthographic camera, which means we’re looking at the scene "straight down" (representing a 2D game - libgdx is also capable of 3D).
2	A `SpriteBatch` lets us draw multiple objects in one pass. Due to how actual games have a lot of objects on screen, using a `SpriteBatch` instance is the standard way of presenting anything on screen in libgdx.
3	Fonts, like many other resources, are explicitly declared for efficiency’s sake. This will give us a "default" font.
4	Once the screen becomes visible, the camera is set to show a top-down view of a rectangle of `[0, 0, 800, 480]` - since this is the same area as the "desktop" target’s window, each scene point corresponds to a single pixel.
5	The `SpriteBatch` is ordered to present the same viewport as the camera.
6	A batch must be always explicitly begun and ended. Objects are queued for drawing with it in between those two points.
7	Here’s where we actually draw the text. The X coordinate corresponds to the far-left side of the screen, and the Y coordinate corresponds to the bottom of the screen plus the capital-character height of glyphs of our font.

This nets us an actual Hello World, but we still have nothing to work on.

Tick tick tick tick

Screen s in libgdx are updated once per an interval called a tick. For each tick, the render method is called, and only once it’s complete can another tick occur^[3]. This means all your game’s logic other than (de)initialization should be called from render, and - ideally - run in the same thread.

Due to machine-specific issues or issues with the game logic itself ^[4], the length of time each tick can take is not constant - this is why the delta parameter is provided, giving the millisecond amount between current an previous ticks.

As you have probably figured out by now, we’re going to need to tap into the tick mechanism to get started. So, let’s end this installment by printing up subsequent ticks. Here’s how our MainScreen will look now:

class MainScreen extends ScreenAdapter {

  lazy val camera = new OrthographicCamera()
  val batch: SpriteBatch = new SpriteBatch()

  var tick = 1L

  lazy val font = {
    val f = new BitmapFont()
    f.getData.setScale(2f)
    f
  }

  override def render(delta: Float) = {
    tick += 1 (1)

    //print tick
    Gdx.gl.glClearColor(0, 0, 0.5f, 1)
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT)
    camera.update()
    batch.setProjectionMatrix(camera.combined)
    batch.begin()
    font.draw(batch, s"Tick: $tick", 0, font.getCapHeight) (2)
    batch.end()
  }

  override def show() = {
    camera.setToOrtho(false, 800, 480)
  }
}

1	Setting the tick number…
2	…and printing it out.

You can grab the source code of the entire project in this state here. Libgdx itself has very comprehehsive documentation, so if you wish to find out more about it, its wiki is a good place to start.

Next up

Now we finally have a starting point for actual development of our concept. In the next installment, which will follow very shortly, we’ll get on to do that - specifically, by creating a simple stream that is able to tap into the tick mechanism of libgdx’s engine.

1. This factory-method-based approach to lifecycle management is quite widespread, and used most prominently in the JVM world by the Android ecosystem

2. In the "personal computer" sense: anything running on an i386/amd64 architecture or similar.

3. In other words, the render method is guaranteed to be called at most once simultaneously - similarly to how receive works for Akka actors. This is also how most JVM GUI engines operate.

4. As corollary to the above, sluggish game logic in the render implementation causes the subsequent call to be delayed.

Pomisos released!

Tue, 28 Mar 2017 00:00:00 +0200

What?

As I alluded to earlier, my free time (or at least the portion dedicated to coding) was recently taken up by a specific project. That project has been finally released: say hello to pomisos.

Pomisos is a desktop productivity app that helps users with elements of the Pomodoro Technique, specifically automating distraction removal (e.g. e-mail clients). There’s more info on the project page, what I’m going to concentrate on here are several thoughts that came up during the app’s development.

JavaFX is/was surprisingly ahead of its time

While the previous "official" Java desktop GUI framework, Swing, was based on the Publisher-Observer pattern, JavaFX is instead oriented around Properties.

Although superficially similar (they also allow listeners to be added), the crucial difference is the concept of "binding" properties to one another, essentially synchronizing them automatically. This allows for essentially writing GUI logic like a set of logic rules, e.g. (pseudocode) :

managePane.visible.bind(option.selected)

where visible and selected are both of type BooleanProperty. What’s more, you can bind multiple properties into complex statements, which essentially allows you to create dataflows that are updated automatically.

Essentially, the logic starts to look like a declarative stream specification, much more maintainable and convenient than the "Listener Hell" of Swing. It’s a shame JavaFX arrived only when classic desktop apps have already started to go out of fashion.

There’s comprehensive support for JavaFX in Scala

If you do want to start developing JavaFX applications, however, there’s several libraries that allow to comfortably code using Scala.

The first one, of course, is scalafx, which simply "Scalifies" the API, and adds a convenient DSL for property bindings. The previous example would now look like:

managePane.visible <== option.selected

Even more rule-like^[1]!

Another thing is FXML development. FXML works pretty much like e.g. Android’s UI XML files - it allows you to separate general UI specification (the XML) from your business-logic code (a Controller class).

To take advantage of that convenience in Scala, there’s scalafxml. What’s especially nice is that integrates Scala DI approaches (like Macwire), to automatically instantiate the Controller classes^[2].

Gremlin-based Graph DBs are too involved for simple projects

Over the last year or so, I’ve been evaluating Gremlin/TinkerPop, specifically the Scala implementation, in a variety of projects. Out of curiosity, I went with it as the DB layer for pomisos, using an embedded OrientDB instance as the storage implementation.

Unfortunately^[3], while the API Gremlin API is almost amazingly intuitive (and the Scala implementation doubly so, big kudos to Michael Pollmeier for his work here), I found the "constant" cost of introducing this kind of abstraction seems to be an overkill for projects with only several, unconected, entity types.

Specifically, the biggest problem was creating generalized DAOs, and attempting to use edges as an Ersatz for trivial relations.

To be clear though: in this case, the aforementioned cost of introduction is comparable to an RDBMs. It’s just that, for projects of scale comparable to pomisos, simple object stores are more than sufficient.

Relaxed approach to development is the way to go when prototyping

From my observations, it appears there’s a persistent myth that experienced developers somehow create perfect code instantly - like some sort of rainbow-excreting unicorns. I find that exceedingly harmful, since it effectively reduces the amount of code that gets out to the public, due to lowered self-esteem of aspiring devs.

By the way, one of the best things in Clean Code is the part where Uncle Bob shows the continuous process of implementing a simple example, starting from a very messy prototype.

This is exactly why I haven’t squashed the ongoing commits. I think it’s a good idea for there to be more transparency in development of similar-scale projects, exactly because of the learning potential in demonstrating the actual coding process.

Coming up

That’s all regarding the development of pomisos. Check out the project if you’re interested.

Next time, we’ll go back to streaming and game development!

1. Although, to be honest, I’m not a fan of the other parts of the DSL, as I find "plain" map/flatMap combinators would be a much better fit here, instead of a tailor-made DSL.

2. There’s one problem - the DI for those classes is handled at runtime.

3. …and as I expected…

Reactive Streams in Scala meet a Game Engine - Part 2

Wed, 16 Nov 2016 00:00:00 +0100

Series' Table of Contents

Unfortunately, just after the first installment of this series, I’ve involved myself in another side-project, which will probably be released soon. But hey, that’s no excuse to at least make a small update in the meantime, to keep the ball rolling, right?

Scope

Desiderata

As mentioned in the first post, the next step should be defining our scope. This basically boils down to making our exploration of the streaming paradigm as efficient as practicable, specifically identifying problems intrinsic to game programming.

One of those is efficient resource allocation. Games need to spawn and remove potentially large numbers of object quickly, and without taxing the memory of the device. For JVM-made games, Android device limits would probably serve as the most prominent example in recent years^[1]. Game engines like libgdx mitigate this problem by employing pools of mutable data structures (which reduces the number of created objects, and hence GC time), among other tricks.

Another issue is smooth rendering of the game’s updates. Enjoyment drops sharply if the game becomes visibly choppy.

A related problem is responsiveness to player input. Smooth FPS gives little consolation if the player remains constantly frustrated while unable to react in time to the unfolding events.

To sum up…

We would like to have a game that:

requires relatively quick player reaction (so e.g. no turn-based games),
has a lot of events happening at any given moment,
and employs a large number of entities to do so.

Recall also one of our initial assumptions: mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions.

First sketches

Obviously, we want some kind of an arcade game, somewhat close in style to a shoot’em up, even drifting into bullet hell territory^[2].

For starters, we want some sort of controllable character, multiple, respawning enemies, and some sort of score meter (which will help us gauge the rate of updates).

Here’s a veeery rudimentary sketch of how the game will look like at the beginning:

Figure 1. No, I’m not a graphic artist, why do you ask?

Of course, we’ll keep adding stuff as we go along, but we already have more than enough work to do.

Up Next

In the following episode we’ll finally see some code. We’re going to define the boilerplate skeleton of our application, and create an Akka Streams source that allows to react to game state updates.

1. Alongside Minecraft’s chunk management.

2. Apologies for linking to TV Tropes.

Reactive Streams in Scala meet a Game Engine - Part 1

Mon, 12 Sep 2016 00:00:00 +0200

Series' Table of Contents

Intro

It’s always interesting to see a new (or newly fashionable) technique being applied to new and unexpected areas, and see whether it makes sense (and what makes or breaks it).

Such is the case with Scala (reactive) streaming and computer game creation. A while ago, I stumbled upon a talk by Aleksandar Prokopec, which planted the seed of inspiration for this blog series. And what set that seed on a path of growth was another presentation, this time by Michał Płachta ^[1].

So, why another thing on this?

Well, of course it wouldn’t be fun if the blog series was just a same-scenario rehash. Here’s what’s going to be different - while the aforementioned talks were more-or-less "from-scratch" affairs, creating a game engine and/or a streaming platform along the way - I will be going the "lazy" route, and attempting to join together a ready-made game engine with a ready-made streaming framework.

The primary advantage of this kind-of-unique^[2] setup is that the focus almost by definition falls on the paradigm mismatch problem, and thus it’s easy to see what one must solve to make the whole shebang work.

And there’s a related blessing in disguise - because we’re operating between two rigid boundaries of stable libraries, any domain-related idiosyncrasies will be easily identifiable, and have a reduced chance of rising up later on to bite is in the hind parts.

Now, let’s talk framework choices.

Decisions, decisions

Game Engine

It may come as a surprise to some of the readers, but JVM game programming is not exactly a barren field, despite the infamous "Java is slow" stereotype. Probably the greatest influence in recent years has been the rise of Android, hence why a high proportion of JVM game engines provide support for both "mobile" and "desktop" targets^[3].

In any case, two engines seem to be the strongest contenders at the moment: libGDX and jMonkeyEngine. Both have been present on the scene for several years, and offer comprehensive game development feature sets^[4].

The choice, however, falls on libGDX, for two reasons: it has explicit support for 2D rendering, and it’s the one I’m actually familiar with.

I’m going to expand on libGDX in upcoming installments. Right now, let’s roll over to the other side for the…

Streaming Implementation

Apart from the obvious choice of Akka Streams, I was also monitoring the progress of Swave.

I recommend following its development closely. It is, in implementation and design, significantly less complex than Akka Streams. Swave can therefore serve as a convenient stepping stone in beating the learning curve, when adapting to the streaming paradigm.

However, Swave is still shaping up, its documentation is somewhat lacking^[5], and I have a sneaking suspicion that some unique features of its "competitor" (like blueprinting) might come in handy, so we’re sticking with Akka Streams.

OK, so the tech base is decided now. All that remains is one tiny detail - just what is the end goal and how to achieve it?

Some ground rules

Let’s start with the goal itself, since it’s pretty obvious - a natural way to verify whether something works (or doesn’t), is through a practical proof-of-concept. Therefore, during this series, I’m going to work towards a fully-playable simple game, that uses streaming as its core operational model.

This leads nicely to the preferred form of the series, i.e. a development blog - meaning:

installments will be posted (hopefully) regularly,
with a soft word-length cutoff,
mistakes will be made prominent (and prominently made), including potentially avoidable ones, in order to showcase them and their solutions^[6].

Finally, to spice things up a bit, I’m going to deliberately forgo gathering "technical" inspiration from the aforementioned related talks and projects (like the additional abstractions introduced by Dr. Prokopec), in order to see where the problem domain lends toward convergent solution creation.

Up Next

In the next installment, we’re going to cover defining the scope of the PoC game. Stay tuned!

1. Source code here.

2. There are at least two similar projects. One went in the direction of providing an Rx* translation layer. It seems to be inactive for about two years now. Another, implemented in Kotlin, seems to be under active development. Both use the Rx* family of libraries.

3. Be aware 'though that since Android supports "native applications", a lot of Android games are written in C/C++ instead.

4. To answer a - perhaps - lingering question: Minecraft, the poster child of JVM games, uses neither. It has a proprietary engine, with several native bridges for OpenGL support etc.

5. Do see the presentations 'though for a good feel of what it offers.

6. Also making for a nice excuse in case of eggregious blunders.

Painless, type-safe config file parsing with Macwire and Ficus

Sat, 30 Apr 2016 00:00:00 +0200

Intro

Scala has a distinct advantage of an "official" configuration file format, i.e. HOCON. Even better, the niceness transcends the officialdom - HOCON is quite well-defined, and it has lots of useful of features. In other words, it’s simply a good format.

Of course, any Scala developer past their Day 3 can tell you of a certain little quirk - the reference parser implementation is a Java library. Even though Java-Scala interop is actually mostly tolerable once one learns the ropes, it still remains slightly annoying.

This post will describe a possible approach for reducing that annoyance by synergizing two well-known libs.

The following is intended for devs already minimally familiar with Macwire. If you’re not, take a look at the README. Reading the the Introduction will suffice.

Config

There are several popular pure-Scala HOCON parsing libraries available. We’re going to use Ficus.

As a library, it’s extremely straightforward - specify the config file, the subpath within it, the type you want it to be parsed into - the lib will do the rest. It even supports extraction into semi-arbitrary data types, like so:

server {
    host: "example.com"
    port: 1024
  }

case class ServerConfig(host: String, port: Int)

import net.ceedubs.ficus.Ficus._
import net.ceedubs.ficus.readers.ArbitraryTypeReader._

ConfigFactory.load().as[ServerConfig]("server")
//emits ServerConfig("example.com", 1024)

Ad Rem

First Approach

So, keeping with the theme, let’s wire up a life support system for plants. Say we have two HTTP services - one for moisture control, and the other handling miscellaneous sensors:

services.scala

class HydroService(val config: ServerConfig) {
    //logic goes here
}

class SensorService(val config: ServerConfig) {
   //logic goes here
}

where, again, the config class is:

case class ServerConfig(host: String, port: Int)

So, our config file could look like this:

app {
  hydro {
    host: "hydro.example.com"
    port: 8215
  }

  sensors {
    host: "sensors.example.com"
    port: 1777
  }
}

To make things sweet and simple, let’s encapsulate the the entire (relevant) config into a container class:

AppConfig.scala

class AppConfig(val hydro: ServerConfig, val sensors: ServerConfig)

case class ServerConfig(host: String, port: Int)

We’ll now take the first stab at the application runner:

Run.scala

import com.softwaremill.macwire._
import com.typesafe.config.ConfigFactory

object Run extends App {

  lazy val rawConfig = ConfigFactory.load()

  import net.ceedubs.ficus.Ficus._
  import net.ceedubs.ficus.readers.ArbitraryTypeReader._

  lazy val config: AppConfig = rawConfig.as[AppConfig]("app") (1)

  import config._ (2)

  lazy val hydroService = wire[HydroService]
  lazy val sensorService = wire[SensorService]

  println(s"HydroService configured with ${hydroService.config}") (3)
  println(s"SensorService configured with ${sensorService.config}")

}

1	It is a good idea for your app’s config to be contained in a dedicated subpath, mostly for reasons of avoiding namespace clashes, but also due to technical difficulties of loading the "root path".
2	Bringing ServerConfig instances into scope.
3	Debug statements.

Obviously, we’ll end up with:

> runMain Run
[info] Updating {file:}root-201604ficusplay...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 3 Scala sources to target/scala-2.11/classes...
[error] src/main/scala/Run.scala:24: Found multiple values of type [ServerConfig]: [List(sensors, hydro)]
[error]   lazy val hydroService = wire[HydroService]
[error]                               ^
[error] src/main/scala/Run.scala:25: Found multiple values of type [ServerConfig]: [List(sensors, hydro)]
[error]   lazy val sensorService = wire[SensorService]
[error]                                ^
[error] two errors found

because Macwire cannot distinguish between the two ServerConfig instances pulled into scope with import config._.

So, what can we do?

Tagging to the rescue

Well, it turns out Macwire possesses the notion of Qualifiers. In intent, they are identical to JSR 330 qualifiers, the only difference is that they operate on types instead of annotations (since Macwire performs its DI based on types).

OK, so we need to qualify our two service dependencies with some sort of tagging types. We could create marker traits for that. However, we notice that we:

already have two distinct types
that unambiguously convey our intent.

I’m talking, obviously, about the service types themselves!

OK, so let’s tag those config dependencies:

services.scala

import com.softwaremill.tagging._

class HydroService(val config: ServerConfig @@ HydroService)

class SensorService(val config: ServerConfig @@ SensorService)

Of course, for MacWire to know where to get these dependencies from, we have to tag the config class as well:

AppConfig.scala

import com.softwaremill.tagging._

class AppConfig(val hydro: ServerConfig @@ HydroService, val sensors: ServerConfig @@ SensorService)

case class ServerConfig(host: String, port: Int)

OK, looks like we’re all set. Let’s run our app now, aaand:

[info] Compiling 1 Scala source to target/scala-2.11/classes...
[error] src/main/scala/Run.scala:20: Cannot generate a config value reader for type
com.softwaremill.tagging.@@[ServerConfig,HydroService], because value readers cannot be auto-generated for types with type parameters.
Consider defining your own ValueReader[com.softwaremill.tagging.@@[ServerConfig,HydroService]]
[error]   lazy val config: AppConfig = rawConfig.as[AppConfig]("app")

Still no go.

Rescuing the tagging

The error message pretty much spells out the problem ^[1]. Since the config instances are now of type @@[ServerConfig,XService], Ficus is unable to find a way to construct instances of their types.

Via the error message, we’re offered a solution of implementing a ValueReader, which is what Ficus depends on when transforming config files into instances. Fortunately, from our previous attempt, we know that Ficus already has ValueReader objects in scope capable of generating case classes like ServerConfig ^[2]. Additionally:

ValueReader has a map method,
MacWire provides a taggedWith[TTag] helper that converts any type TType into @@[TType, TTag].

Let’s put those pieces together and add our custom reader for tagged types:

implicit def taggedReader[TType, TTag](implicit reader: ValueReader[TType]) = reader.map(_.taggedWith[TTag])

After we add the above to run, we should end up with:

> runMain Run
[info] Compiling 1 Scala source to target/scala-2.11/classes...
[info] Running Run
HydroService configured with ServerConfig(hydro.example.com,8215)
SensorService configured with ServerConfig(sensors.example.com,1777)
[success] Total time: 2 s, completed 2016-04-30 15:08:12

And we’re pretty much done!

One final improvement that we can make is to better convey the relationship between the transformed type and ValueReader. We do this by using the following equivalent form of our reader:

implicit def taggedReader[TType: ValueReader, TTag] = implicitly[ValueReader[TType]].map(_.taggedWith[TTag])

In the end, our app class looks like this:

Run.scala

import com.softwaremill.macwire._
import com.softwaremill.tagging._
import com.typesafe.config.ConfigFactory
import net.ceedubs.ficus.readers.ValueReader

object Run extends App {

  lazy val rawConfig = ConfigFactory.load()

  import net.ceedubs.ficus.Ficus._
  import net.ceedubs.ficus.readers.ArbitraryTypeReader._


  implicit def taggedReader[TType: ValueReader, TTag] = implicitly[ValueReader[TType]].map(_.taggedWith[TTag])

  lazy val config: AppConfig = rawConfig.as[AppConfig]("app")

  import config._

  lazy val hydroService = wire[HydroService]
  lazy val sensorService = wire[SensorService]

  println(s"HydroService configured with ${hydroService.config}")
  println(s"SensorService configured with ${sensorService.config}")

}

Summary

We’ve managed to get Macwire and Ficus happily working together to parse HOCON config files, in a type-safe manner.

Note that creating a master config object in larger, multi-module projects, is a Bad Idea™. However, that’s not a problem, since you can just pass on the "raw" config (from typesafe-config) onto the relevant submodules, and have Ficus generate the actual config instances there.

Overall, this approach scales well (architecturally), and requires only the minimal boilerplate necessary for type safety.

You can find the full code example on GitHub.

1. If only more Scala libs took this degree of care with compile-time messages…

2. Since the original approach went pass the "Ficus stage" successfully, and only emitted an error during wiring.

Learn AND remember with Anki: Part 4: Statistics and Conclusions

Wed, 26 Aug 2015 00:00:00 +0200

Previously

We’ve just went over a practical example of the scheme. Now, let’s talk stats and conclusions.

So how does it work out, really?

While I can’t offer you any peer-reviewed, double-blind study on the validity of the method I’ve described above. Subjectively however, I can tell you that I am convinced adopting the technique has had a great effect on improving the time effectiveness of learning stuff I want to know. Without fail, for every such source, whether a book, a course, technical documentation or otherwise, I have progressed beyond just that "feel-good" sensation and actually was able to recall the most important information many, many months after first absorbing it.

However, I do have some objective, if not statistically representative, data. Take a look at the following figures, which describe the total time and total number of card reviews throughout my entire usage history, with my ~850 card collection, grown over time:

Stats since day 1. Note the "Average for days studied" entry.

As you can see, the amount of time per day, even if artificially inflated by the aforementioned commuting, is laughably small when compared to the benefits. The benefits themselves, in turn, are indirectly demonstrated by the portion of "relearn" (i.e. "I forgot all about it") card reviews in the overall scheme - Anki/SRS really is that effective in helping you remember ^[1].

In closing

It would be a disservice to Anki if I’d fail to mention that it is also an excellent tool for helping in learning a language. Anki was, after all, originally created to aid its author in studying Japanese.

In fact, a comprehensive collection of community-created decks has been made available, spanning not only multiple languages but also subjects such as Biology, Geography, Physics, and others.

Finally, I hope I have shared some of the enthusiasm for Spaced-Repetition-assisted learning in the context of technical knowledge ^[2]. I encourage you to try it out for a small dataset, and see how it goes - as long as you have a minimum of self-control and curiosity, you should benefit from adopting this technique. Maybe it won’t help you win "Jeopardy!", but it will certainly provide you with a powerful tool when dealing with the ever increasing corpus of information that must be absorbed in order to stay up to date, and to keep in touch with the "core" knowledge that serves as the foundation for your daily technical decisions.

1. A recent example - after preparing a simple deck, and some minimal practice, I was able to write complex MongoDB aggregations off the top of my head, many months after completing the related online course.

2. and haven’t bored you to tears while at it.

Learn AND remember with Anki: Part 3: From theory to practice

Tue, 4 Aug 2015 00:00:00 +0200

Previously

During the last episode, we’ve discussed a learning scheme that can help you to avoid having most acquired information from floating away.

In what follows, we’ll use an example to walk through the scheme step by step.

Intended audience

While the bulk of this series offers information that should be useful for virtually everyone, the examples below are tailored towards programmers, specifically JVM polyglot programmers. You have been warned.

Enough theory, time for practice

Step 0: Choose what to learn

For this article, I’ve decided to use the example of ScalaTest matchers. That topic makes a good example, because:

the subjects squarely in the category of "in one ear, out the other" ,
remembering brings a tangible benefit - reports for the specialised matchers are more meaningful than those from a simple should be equal.

Let’s start, shall we?

Step 1: Highlight Key Passages (Optional)

The matcher documentation (again, available here) is actually a good case where this step can be skipped. The docs are compact, information-rich, and the most notable facts are plainly visible.

Step 2: Create Mind-map-like notes

Before we start, if you don’t have a favorite mind-mapping application, download and install Freemind.

First of all, we’re not actually going to do the full docs, just enough to create a relatively good example. So, let’s do the first 3 sections, up to and including "checking strings".

Here’s how a note-centered mind map might look like:

The resultant mind map.

You might notice the following things:

consistently with "normal" mind maps, items progressively become less general and more specific the further away from the center,
also like "normal" mind maps, items connected "physically" are also related semantically ^[1],
unlike typical mind maps, there very little formatting and decorations were applied. This is because the map is intended to be a transitory step, and not something you will revisit often. Normally, the only formatting I do is hyperlinks and icons for denoting especially important stuff.
only a portion of the information from the doc page is included. It’s a deliberate decision - the goal is to provide a source for quickly writing up cards, which are supposed to help you in the long-term - so we’re looking for stuff that’s most useful. In other words, we need the info for your regular life/work, and not to prepare for a fire-and-forget exam.

Step 3: Extract flashcards from the mind map in Anki Desktop

If you haven’t done so already, download and install Anki on your computer.

First, click Create Deck ^[2]. Since I already have a Scala deck, I’m going to make a subdeck, which is done by writing Scala::ScalaTest. After the deck is created, click on it.

Ready to add cards.

Now, we’re ready to create the cards proper. Click on Add at the top ^[3].

New card to be filled out.

You’ll be presented with a view similar to the above (the additional UI elements mostly come from the source code plugin).

We’re going to out of order and start with the second point - asserting size and/or length - since it’s more straightforward. Let’s start with the length:

Front: I recommend this to be titled in the way that is the most similar to how you’d try to recall the relevant information. In this case, I’d write "ScalaTest - asserting length".
Back: this should contain the relevant information in the most concise way. In this case, all we need is the text of the assert, e.g. have length X ^[4].

Card filled out.

Now, do the same for asserting size. Here you might think whether it wouldn’t be more efficient to store both pieces of information in a single card. In practice, I find that it’s much more worthwhile to split information into as many cards as possible, as simple cards are much easier to learn and hence produce a better retention effect.

Going up next, we have the string comparisons. Here, I’d suggest to create 8 cards:

3 per each "simple" matcher
4 per each "regex" matcher
1 for the withGroups matcher qualifier.

Simple comparison, note slogan-like front naming.

After that, we’ve got comparisons. Here, I’m going to create just two cards, one per every strict/equal-or pair, since the cases appear to be both semantically and "graphically" coupled.

Two pieces of info, but still pretty straightforward.

Note that here I’m omitting the Ordering[T] information, as I’m making the assumption that I can rely on the compiler for recalling that tidbit. However, if you e.g. use lots of custom classes with dedicated Ordering implementations, creating an additional card might be worth your while.

Finally, let’s go back to the general equality case. The be card is pretty straightforward:

Simple equality.

However, for the equal we have to cram some info on the back:

The most complex card type you should be making.

This is unfortunate, however sometimes you have pieces of information that are inseparable.

And there you have it! If you want to study using the cards created for this section, download the pack from here.

Step 4: Learn from flashcards in the Anki Mobile Client(s)

Now that we’ve created the cards, we can move to actually using them. For that, you’ll need to:

Get the mobile client, in either the Android or iOS versions
Set up AnkiWeb synchronization between your desktop client and your mobile device.

Please be aware of the content policies before you sync data through AnkiWeb.

Now that you’ve set it up, and open the synced deck, you should see something like the following:

Before…

…and after the first answer.

You’re viewing a new card in the learning stage, which will require at least one repetition on the same day. Always choose the option that authentically reflects your recall ability of that card. Eventually, you should arrive at something similar to the following screen:

Last answer for now. Note the different time periods.

Now the choices extend beyond the same day. If your recall is increased effectively, the interval selection will lengthen at a geometric rate. So when in doubt, answer pessimistically - it won’t "cost" you much time.

Use the Widget

If you’re using Android, the widget is a nice feature. It shows you the amount of pending cards for the day, as well as the projected time required for going over them.

Step 5: Review, Heal, and Grow your flashcard "Deck"

This is probably the most personalized of the steps. However, a couple common issues might manifest, most notably:

Errors on the cards

Factual, typographical or otherwise: minor ones can be remedied immediately through the mobile client, but for more complex stuff, your best bet is to mark the card by favoriting it, and potentially burying the card, "postponing" it to the following day, if the problem is severe enough so that you can’t meaningfully review the card ^[5].

Hide/Delete will show Bury.

Lack of perceived benefit from using the cards for a particular topic

This mostly stems from not enough understanding of the subject matter. With this method, always try to make a conscious effort to actually comprehend and integrate the source material. This even applies to our relatively simple example - the retention effect for the ScalaTest will be much greater if you practice writing the matcher expressions during the the initial days.

General frustration

In other words, the bog standard reaction to a new habit that one is trying to form. After the initial enthusiasm dwindles, the act of repeating the cards might appear like a chore. In this case, I advise you to

simply stick it out if you’re only a couple of weeks in,
be honest - if you forgot the card’s content, choose "Again",
otherwise try to remember situations where the use of this technique has improve recall, and
take note of how much time have you’ve already spent by using this method ^[6].

Recalling knowledge you no longer need

It happens, whether when studying for actual exams, investing in learning a technology that didn’t pan out, or for other reasons. In this case, you just need to suspend the relevant cards, causing them to no longer appear until they’re manually resumed.

Amassing a huge backlog

To prevent that, start small (<100 cards total) until you get the gist of it, and try to set aside some regularly scheduled period in your daily routine for the card review - for example, during your daily commute ^[7].
To resolve it, just chip away at the mass of cards for several days. Due to how the SRS algorithm works, you are bound to eventually reduce the backlog to a manageable size.

Coming Up

In the final part of the series, we’ll look at some actual statistics related to using Anki, and touch on some miscellaneous closing points.

1. There are no additional connections in this particular example, but, in general, it’s OK to make them.

2. A deck is simply a collection of Anki cards.

3. Yes, the Anki UI is sometimes all over the place.

4. Pardon the faint red markings on the illustration - there seems to be a bug in the Anki Linux client that makes disabling the spellcheck impossible, hence a quick photoedit job.

5. Often happens with formulas encoded in LaTeX, as the mobile client is notoriously fickle when displaying them.

6. Yes, I’m basically suggesting an abuse of humanity’s susceptibility to the Gambler’s Fallacy.

7. Which was how I started.

Learn AND remember with Anki: Part 2: Putting it all together

Tue, 23 Jun 2015 00:00:00 +0200

Previously

In the starting article, I’ve provided a quick overview of what Spaced Repetition means, and talked about the tools that can facilitate it. Now, let’s see how we can put these tools to good use.

The Scheme

Based on using Anki for knowledge retention, I’ve developed a learning scheme to keep myself from forgetting the stuff I’d rather remember. Given a new source material (e.g. a book, an online course), here’s how it looks like.

Step 0: Choose what to learn

Probably the most important step - you need to consciously, and responsibly, ask yourself the question: "what knowledge do I need to retain effectively"? This is something that you optimally should work out for yourself.

However, I can provide three pointers:

The best fit should align with one of the two categories I’ve described in the introductory sections.
You already need to understand whatever you’re trying to recall - the scheme I’m describing want help you with that.
Don’t start with adding "feel-good stuff" - knowledge that you always wanted to recall off the top of your head in order to impress yourself. You’ll most likely, eventually get frustrated and see the exercise as a waste of time. Do begin with applying this learning method to things you actually need.

Step 1: Highlight Key Passages (Optional)

This is of course only relevant for textual sources. I still recommend it whenever you’re learning from those, because it helps you stay in the flow.

Step 2: Create Mind-map-like notes

Regularly, for each piece of content (video for online courses, chapter for a book, etc.), add notes to a mind map describing the thing you’re currently learning about ^[1].

This may seem like a superfluous step, but it helps in three things:

it actually helps with knowledge retention by preliminarily organizing it,
it makes the next step go faster, since you don’t have to re-read/-watch the entire content,
most importantly, it helps to verify whether you’ve understood what you’re learning about. You will very likely to see that you’re e.g. writing nonsense at this point, and it’s less frustrating to identify it and fix it than in the next step.

I say "mind-map-like" because someone might argue that the notes might not necessarily look like a "proper" mind map, i.e. there are no colors, graphics, etc. Nevertheless, I think the form of a tree, which mindmaps take, is the most conducive to general, rapid note-taking.

For the actual mind mapping I prefer to use Freemind. It’s open source, free, multiplatform, and offers a set of convenient keyboard shortcuts for quick note-taking.

A mind-map example - an (outdated) mind map of Akka documentation.

Step 3: Extract flashcards from the mind map in Anki Desktop

Now that you have rapid-access notes, it’s time to go through them and create the flashcards. We’ll mostly be dealing with "classic cards", i.e. with two faces - an entry, and whatever you should recall according regarding the entry. However, there are also other types: reversed (where the "entry"-"recall" relation is bilateral, effectively doubling the given card), and cloze completion, where you "fill" in text (useful for e.g. learning phrases in foreign languages).

Also, you can insert almost everything into a card, including:

plain and formatted texts,
pictures,
sounds,
LaTeX, including complex formulas in math mode,
source code, with the correct plugin ^[2].

See the figure in the introductory section for some examples.

Step 4: Learn from flashcards in the Anki Mobile Client(s)

Reviewing a card in Anki.

Whereas the Desktop client is more conducive to creating (and reviewing) cards, I find it much better to actually learn by using the Anki software on mobile devices (iOS/Android ^[3]).

Principally, this is because, with the mobile client, you have less distractions. And trust me, during the beginnings, once the initial enthusiasm wears out but before you see tangible benefits, you will need all the focus you can get.

Another, equally important thing is that you will have access to the cards wherever you go. This makes it an easier to form the habit of reviewing the cards daily, especially in the case of the Android client, where you can also add a widget to your home screen.

Step 5: Review, Heal, and Grow your flashcard "Deck"

Your cards, once created, should not be treated as if set in stone. Splitting a card, improving it, suspending it, or removing it altogether should all be options you need to consider.

Coming Up

In the next installment, we’ll go from theory to practice and see how it pans out.

1. I usually keep a map e.g. per book.

2. Did I mention Anki has a large plugin directory?

3. While the Android version is completely Open Source, the iOS version is paid-for, and, in fact, the latter is currently the main source of monetary support from the project.

Learn AND remember with Anki: Part 1: Introduction

Tue, 16 Jun 2015 00:00:00 +0200

Intro

Have you ever:

regretted that you only remember some bare trivia from a course or a book that you studied several months ago? Or maybe even stuff from university/college that you wish you’d recall now?
constantly annoy yourself with looking up some implementation detail, API definition or another fact, that you need every couple of weeks, but juuuust manage to slip your mind the next time you require it?
feel like learning new stuff (perhaps to progress beyond your current job) is a Sisyphean task that doesn’t net you anything?

If so, than this series of articles is for you.

The Perils of Ad-Hoc Learning

The nice thing about living now is having slightly more free time for your basic non-survival needs. The absolutely scary thing, in turn, is that the amount of information you have access to, and can gain, is simply overwhelming. Understandably this becomes even worse for IT professionals, what with a new library/framework coming out every week ^[1].

Of course, a lot of this information is fire-and-forget. Another portion you use day to day and, by this virtue, remember without any problems.

What’s left can be divided into two broad categories.

One is knowledge that you use in irregular, but relatively frequent intervals. An example would be an API quirk that you keep re-reading about every couple of weeks. Irritating, isn’t it? Also wasteful.

The other contains "core" knowledge - stuff that you’re not necessarily using directly, but nevertheless benefit from recalling it readily. Forgetting this kind of information is much more insidious - you just end up doing stuff less effectively; or, perhaps one day you suddenly realize that you completely forgot all the things you wanted to learn from the course you did half a year ago.

But, but, the Internet!

A commonly raised counterargument against investing time in recall methods boils down to:

Why bother? Everything is on the Internet anyway, if I don’t remember something, I’ll just look it up!

And sure, you can do that. But how much does it cost you?

Here’s an example: remind yourself of the last small fact or info tidbit that you required and you needed to search for. Now, take your favorite stopwatch app ^[2], and try out finding the answer on the Internet.

I’d be willing to bet that for most of you, the total time between switching away from this article, and returning to it after searching will be slightly north of 10 seconds.

That’s not a lot, right? But those search bits add up during the day, and you need to factor in the context switch costs ^[3].

Now try to recall the answer again, from memory. You, quite likely, still can, and it probably took you less than a second to remember. That is an order of magnitude of a difference. Making a somewhat strained analogy, it’s quite similar to the difference between reading from disk and reading from RAM.

Of course, you’re still spending time by putting additional effort into knowledge retention - but that’s dedicated time, as opposed to hacking up your work routine into disjointed bits.

Spaced Repetition - a solution

Obviously no one is going to sit around every day working on remember their cumulatively growing knowledge by rote.

But there are shortcuts to do something very similar much more efficiently.

The concept of Spaced Repetition offers one such shortcut. There are many implementations exploiting the idea, but they all boil down to taking advantage of particularities of the human brain in order to achieve effective recall of various facts and concepts, with relatively minimal effort.

An SR-based approach

One such implementation is Anki. It’s flashcard-based spaced repetition software. Its documentation can be found here, and it’s available for download on:

The tl;dr version of the process of working with flashcards and SRS looks like this:

you create a flashcard. In the simplest version it’s a note with two faces: the prompt for what you want to learn, and the thing you want to learn. Here’s a couple of examples.

A selection of Anki cards: normal text, source code, image, LaTeX formulas.

after several initial repetitions, Anki prompts you with the cards, in increasing time intervals: initially a couple of days, then a couple of weeks, months, and so on. At each repetition, when viewing the answer, you are prompted to choose one of the following options:
- "Again" which means you forgot about the entry, and need to reset the learning process,
- "Good" meaning you have good recollection, i.e. the vanilla option,
- "Hard" implying thatas you’ve sorta learned, but aren’t quite confident, leading to a shorter time to repetition,
- "Easy" that tells the system to extend the time interval to a greater extent than with "Good".

As you have probably figured it out, the biggest benefit from this system is that the time intervals are managed automatically, and you get automatic reminders to repeat your cards (which is especially useful if you install the mobile client).

Local Flavor

Anki’s repetition selection algorithm is a derivative of SM2, which was invented in Poland.

Coming Up

In the second episode of this series, we will talk about a learning scheme developed empirically by Yours Truly, that takes advantage of Anki when acquiring technical (and other) knowledge.

1. and a Javascript implementation following up in 5 days.

2. I expect very few people still have actual stopwatches, digital or analog.

3. Also, let’s not kid ourselves, you probably took this as a challenge and done the search than you would normally do under this kind of situation.

Musings on Ansible

Thu, 9 Apr 2015 00:00:00 +0200

This is a set of loosely-related remarks and observations, accrued due to recent use of Ansible in several work and personal projects (and on the latter note, I made a thing).

The entry is mostly intended as a writing exercise, so if you’re evaluating Ansible at the moment, please keep this in mind.

A word of Introduction

Ansible, like Chef, Puppet and Salt, is configuration management software. Despite my best intentions to reach the widest audience possible, providing a comprehensive overview of the field would require creating a whole series of blog posts.

Therefore, wanting to convey the following statements in reasonable time, I’m forced to limit the target readership to those that posses basic knowledge of the subject. In other words, if you are unfamiliar with the subject, please do take your time to browse the included links. Sorry!

Ansible - the good, the bad, the ugly

Context

All those remarks are made by someone who is strictly more a dev than an op, so the things I’m writing about here may or may not apply to your situation as much as they did to mine.

Also, the entry was writen when going through the Through of Disillusionment, halfway to the Slope Enlightenment, and therefore may sound more negative than intended.

Good: Workerless setup

The absolutely wonderful thing about Ansible is its agentless architecture - you don’t need to setup anything on the services machines, other than a valid SSH connection to a user (usually with possible sudo access). No special nodes requiring additional setup (like in Chef), nothing of the sort.

This in fact makes it very convenient to bootstrap setups for CI and the like, or even a quasi Inversion-of-Control setup, using the ansible-pull utility.

Such a feature may seem like a small thing, but it reduces the error rate during the "metagame" of setting up your servers.

Good: Easy to understand syntax

Ansible uses YAML with embedded Jinja2 for its configuration definitions, and most of the basic stuff can be expressed that way. Getting the majority of desirable output defined is pretty straightforward, once you learn the basics.

Bad: …that’s sometimes not as intuitive as it should be

A big stumbling block I’ve encountered is correctly specifying the conditions in when blocks (saying when to execute a task) and similar ones. Truth be told, even after viewing the parsing source code for the "playbooks", I’m still not entirely confident on what is and isn’t allowed.

I think this is due to the fact that the Jinja2-based syntax sits pretty squarely in the Uncanny Valley for someone with an off-and-on Python background. In effect, you end up writing those conditions like in Python, which works… about 90% of the time. The remaining 10% will piss you off to no end.^[1]

Good: Has a strong focus on idempotency

Pretty much a given for all modern configuration management software, but nevertheless I could excitedly rave and rant about that at you the entire day - it’s awesome that Ansible specifically focuses on what the state should be, rather than what tasks to do.

Good: Trivial to customize

To recap, here’s how config management works in Ansible:

the unit of work is a task, meant to encapsulate a single "end-state" quantum, i.e. something that should be ensured to be fulfilled once this task is done.
tasks use modules, which do the actual grunt work and can be implemented in most languages (a lot of them use Python, obviously since Ansible is written in that). There exists a cornucopia of Ansible built-in modules, from user management through ensuring a given is in a file, to EC2 instance setup.
tasks can be grouped into roles, which can also contain common variables, custom modules etc..
task and role mixes are codified into playbooks, which describes what your actual configuration should look like. You "run" the playbooks on your "inventory"^[2] to achieve that desired state.

Creating roles

By the way, if you’ve dug through the role documentation and wondered how to create the role scaffolding automatically, here’s how you can do that:

ansible-galaxy init <rolename>

This is of course mentioned only in the Ansible Galaxy documentation, later on.

You’ll notice that such a structure allows you to modularize your configuration management logic as you see fit. That includes creating several "internal" modules for complex actions and sticking them into your roles.^[3]

Bad: …but no code reuse for Python modules

Funilly enough, due to how Ansible is structured, you cannot have "common" code files for Python-based modules.

That sucks, but is probably an edge case for most stuff.

Good: Good introductory documentation

The docs ^[4] do a very nice job of showing you the ropes. They are very much example-based, and subsequent steps build upon previous knowledge in a logical way.

Ugly: …with no formal sections

I fail to see, in the documentation, a "Big Picture" overview of how the various components of an Ansible definition are structured.

Apparently, looking at the the latest Stack Overflow poll, I’m in the minority when it comes to education and this may skew my perception, but I would give my right spleen for an EBNF or similarly-formatted formal specification of the Playbook syntax.

Good: A globally shared role repository

Ansible has something called Ansible Galaxy, which is its way to share you common configuration functionality.^[5]. It’s pretty easy to use if you just want to find something (the thing I made is also there).

Ugly: …very nascent in the current state

However, especially if you display perfectionist tendencies ^[6], you will spend quite a bit of time examining the roles for the functionality your require.

One problem is that, while a rating system exists, it’s severely underused, and you have to fish for the well-made roles.

Another is the presence of very clearly clashing philosophies when designing roles. It’s best demonstrated when comparing ansible roles from Jeff Geerling and the Stouts group.

The former offer a definitely correctly functioning role, but make a metric f-ton of assumption about what a given piece of setup software is going to operate as. The latter, meanwhile, while not being entirely correct idempotency-wise, allow for very diverse configuration variants.

To be honest, I came to prefer the Stouts roles for their ability to set up as I want them over the geerlingguy ones, despite Mr. Geerling’s foray into book writing on Ansible^[7].

Overall

The general picture that I’ve painted hopefully shows a framework with a number of nits that you can pick, but built on solid foundations nevertheless. Those solid foundations will provide a payoff as the framework grows and matures, eliminating the smaller problems along the away. Be aware of the shortcomings, but rest assured that I recommend you check out Ansible for you configuration management needs.

1. Of course, for someone working primary in Python, this may not be a problem, due to their probable contact with and prior use of Jinja2.

2. A list of target servers with some labelling and variables.

3. Normally, modules are quite a "big thing", able to be shared stand-alone, but sometimes the convenience of writing in an actual Turing-complete programming language is too great to miss.

4. …which admittedly took a nearly-non-trivial amount of time for me to fish out out of the sales-pitch-filled landing page….

5. If you’ve never used configuration management software, think of it as the equivalent of the Docker Hub Repository.

6. like Yours Truly

7. Can’t comment on the book’s quality, but the roles show good craftsmanship.

Comments added

Sat, 6 Sep 2014 00:00:00 +0200

Against my better judgment I have added commenting functionality to the site. Hopefully there won’t be much spam - went with IntenseDebate for this reason. It supposedly provides better built-in spam-fighting capability.

Frist post

Sun, 31 Aug 2014 00:00:00 +0200

Time for an inaugural post. Regarding the present and future contents, there’s more info on the About page.

As you maybe noticed by the header, this page is generated with JBake. It’s also taking advantage of the JBake Gradle plugin by Cédric Champeau, introduced here (incidentally also reminding me to add a Creative Commons license declaration).

Mikołak

Making LLM coding assistants create graphics…​ and physical objects

Intro

Warmup: Making infographics

Main course: Prompts to objects

Intro

Smoke tests

Baselines

Local model selection

Setup

From matrix multiplication to plastic strings

Practical prints

Design

Implementation

Results

Code Review

Outro

Extracting Machine Learning Training Data from a Video Game – Using "Ready-Mades"

Starting words

Modern Detectiving with Open-Vocabulary Models

Intro

Implementation

Local Large Models to the rescue?

Moondream: initial approach and examination

"Prompt engineering" of input images

Conclusions for Moondream

Do we even need the designator? Looking into auto-segmentation via SAM 2

"Mainstream" LLMs

Claude

ChatGPT

Summary

Extracting Machine Learning Training Data from Video Games – UI element analysis for model training preparation

Intro

Getting started

Designating the designator

Summary

Extracting Machine Learning Training Data from Video Games – Problem Statement

Introduction

Possible premises and their choice

Identifying screen UI elements relevant for machine learning model training

Summary

Combining Python assignment expressions and destructuring: Having your variable and eating it too

Intro

Schrödinger’s destructuring

Collapsing the wave function

Blog update

Travesty 0.9 now available!

Travesty 0.8 now available!

Travesty released!

What?

Where to get it?

Reactive Streams in Scala meet a Game Engine - Part 7

Series' Table of Contents

In this episode…​

Getting input…​ in

Interlude: A Failure to Communicate

The Problem

Finding the solution - source analysis

Finding the solution - debugging

The buffet

Getting to the point

Alternatives?

OK - what the hell?

Making the input count

Discussion

What’s next?

Reactive Streams in Scala meet a Game Engine - Part 6

Series' Table of Contents

In this episode…​

Modularizing the game’s logic

Introducing more thread safety

Using materialization

Assigning a custom dispatcher

Loose ends

Reactive Streams in Scala meet a Game Engine - Part 5

Series' Table of Contents

In this episode…​

A bit of refactoring

Magicks!

Reflowing!

Making LLM coding assistants create graphics… and physical objects

In this episode…

Getting input… in

In this episode…

In this episode…

In this episode…

`Source.actorPublisher`

`Source.actorRef`

In this episode…