Programming Posts · HookRace Blog

Workload Capture & Replay

2026-02-10T00:00:00-05:00

When customers hit issues in production, it can be an effort to locally reproduce them, especially when external sources are involved. Reproducing issues is useful not just to figure out the root cause, but also to verify the fix and add a regression test. The newly introduced workload capture & replay tooling records a Materialize instance’s state as well as recent queries and ingestion rates, then replays them in a Docker Compose environment with synthetic data. In this blog post I’ll show how it works and talk about some of the challenges and future work.

Read the rest of the blog post over on the Materialize blog.

Speeding up Materialize CI

2025-08-08T00:00:00-04:00

In the previous post I talked about how we test Materialize. This time I’ll describe how I significantly sped up our Continuous Integration (CI) Test pipeline in July, especially for pull requests that require a build and full test run. The goal is to make developers more productive by reducing the time waiting for CI to complete.

We always kept CI runtime in mind, but it still slowly crept up over the years through adding tests, the code itself growing larger, as well as hundreds of minor cuts adding up.

Read the rest of the blog post over on the Materialize blog.

How to Use the Materialize Emulator

2024-10-16T00:00:00-04:00

In our last blog about our Quality Assurance (QA) team, we gave an overview of the QA process, including our software and testing methods. One of our key tools during testing is the Materialize Emulator, a Docker image that allows you to maintain a locally hosted version of Materialize.

Read the rest of the blog post over on the Materialize blog.

The CI Flake

2024-08-16T00:00:00-04:00

I analyzed a flaky test failure in our Materialize CI today:

$ docker compose up -d --scale default=0 default
no such service: default
mzcompose: error: running docker compose failed (exit status 1)

I had seen this error already once or twice in the last year, but it was incredibly rare in our Continuous Integration (CI) runs, and never happened locally. As usual, there were more pressing product issues to debug, so I never looked into it. But last week I switched most of our CI tests to run on Hetzner Cloud instead of AWS to save some money. Suddenly this issue started occurring more often in CI, so my thinking was that it must somehow be timing-dependent.

Before investigating my first instinct was that we are not writing the docker-compose.yaml file properly, which had already led to flaky calls to Docker Compose before (source code):

file = self.files.get(thread_id)
if not file:
    file = TemporaryFile(mode="w")
    os.set_inheritable(file.fileno(), True)
    yaml.dump(self.compose, file)
    self.files[thread_id] = file

So yesterday I added a file.flush() after the yaml.dump and hoped to be done with it. This morning I woke up and the issue was still occurring!

Based on the logged docker call this should be the code causing the problem (source code):

def handle_composition(
        self, args: argparse.Namespace, composition: Composition
    ) -> None:
        if args.workflow not in composition.workflows:
            # Restart any dependencies whose definitions have changed.
            # This is Docker Compose's default behavior for `up`, but
            # not for `run`, which is a constant irritation that we
            # paper over here. The trick, taken from Buildkite's
            # Docker Compose plugin, is to run an `up` command that
            # requests zero instances of the requested service.
            if args.workflow:
                composition.invoke(
                    "up",
                    "-d",
                    "--scale",
                    f"{args.workflow}=0",
                    args.workflow,
                )
            super().handle_composition(args, composition)
        else:
            [...]

Running the test in an endless loop locally had no success of reproducing the issue:

while true; do
    bin/mzcompose --find skip-version-upgrade down
    bin/mzcompose --find skip-version-upgrade run default || break
done

I modified the conditional in the above Python code to be if True: and surely that reproduces the issue:

$ bin/mzcompose --find skip-version-upgrade run default
$ docker compose up -d --scale default=0 default
no such service: default
mzcompose: error: running docker compose failed (exit status 1)

So how is it possible to be landing in this code path sometimes? I checked how the composition.workflows are generated and it looks quite complex indeed (source code):

mzcompose_py = self.path / "mzcompose.py"
if mzcompose_py.exists():
    spec = importlib.util.spec_from_file_location("mzcompose", mzcompose_py)
    assert spec
    module = importlib.util.module_from_spec(spec)
    assert isinstance(spec.loader, importlib.abc.Loader)
    loader.composition_path = self.path
    spec.loader.exec_module(module)
    loader.composition_path = None
    self.description = inspect.getdoc(module)
    for name, fn in getmembers(module, isfunction):
        if name.startswith("workflow_"):
            # The name of the workflow is the name of the function
            # with the "workflow_" prefix stripped and any underscores
            # replaced with dashes.
            name = name[len("workflow_") :].replace("_", "-")
            self.workflows[name] = fn

Since we are reading and analyzing the mzcompose.py file, something might be running in parallel and corrupting/overwriting it. If I empty the file I can indeed reproduce the issue with the original code:

$ echo > test/skip-version-upgrade/mzcompose.py
$ bin/mzcompose --find skip-version-upgrade run default
$ docker compose up -d --scale default=0 default
no such service: default
mzcompose: error: running docker compose failed (exit status 1)

It feels like I’m getting closer, but nothing should be running in parallel, even in CI each agent is running in isolation, the git pull is long finished at this point based on the CI logs, so it should not interfere. As an additional frustration, I’ve been wondering why this flake is mostly (but not only) occurring when running this test? The mzcompose.py file in question is one of the simplest ones around.

Until here it felt like I was getting closer, but now I had no more leads. I went as far as searching for Python bugs around inheritance that could explain this issue, but if Python was buggy in such basic behavior, this code definitely wouldn’t be the first one to run into it. Could it be a cosmic ray?!

Since this issue seemed unexplainable without the universe conspiring against me, I went back to the basics and checked what else is running in CI and I found something extremely suspicious just before the actual test execution in our CI script (source code):

# Start dependencies under a different heading so that the main
# heading is less noisy. But not if the service is actually a
# workflow, in which case it will do its own dependency management.
if ! mzcompose --mz-quiet list-workflows | grep -q "$service"; then
    ci_collapsed_heading ":docker: Starting dependencies"
    mzcompose up -d --scale "$service=0" "$service"
fi

Oh no! There is another $service=0 call here, my entire investigation so far might have been useless! From looking at the CI output it’s not clear which one is actually running, but the fact that I’ve never seen the flake locally points to the new suspect. After all this code is only running in CI, and not locally, all in the name of cleaner CI logs! Checking the CI output there is a strange line occurring:

write /dev/stdout: broken pipe

Well, there is a pipe between the mzcompose and grep processes, but why would it break? Let’s run the mzcompose command in isolation:

$ bin/mzcompose --find skip-version-upgrade list-workflows
default
test-version-skips

It lists all workflows, of which we have two. The grep command is only looking for the default line. Apparently, grep is smart enough to exit immediately when it finds a match when used with -q (manpage):

-q, --quiet, --silent
       Quiet; do not write anything to standard output. Exit
       immediately with zero status if any match is found, even if
       an error was detected. Also see the -s or --no-messages option.

Easy enough to verify, I ran the true command, which exits instantly:

$ bin/mzcompose --find skip-version-upgrade list-workflows | true
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

So this broken pipe output is indeed explained by grep exitting early. But why does the mzcompose process breaking lead to the code in the if’s body being executed? Normally the first command in a pipe failing would be ignored, since the second command succeeded, and I’m seeing the return code being 0 here. Checking the beginning of the script reveals that we use the pipefail setting:

set -euo pipefail

Bash’s manpage explains:

The return status of a pipeline is the exit status of the last command, unless the pipefail option is enabled. If pipefail is enabled, the pipeline’s return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully.

Typically, Python buffers its output when it doesn’t write directly to a terminal, so I still can’t reproduce the issue with grep locally. Then I remembered that we had an issue with our Python tests’ stdout and stderr output being garbled, which made us enable PYTHONUNBUFFERED in our ci-builder Dockerfile:

# Ensure that all python output is unbuffered, otherwise it is not
# logged properly in Buildkite
ENV PYTHONUNBUFFERED=1

With this revelation the flake was easy enough to reproduce. Just running this would fail occasionally, still less than 1% on my system:

$ set pipefail
$ PYTHONUNBUFFERED=1 bin/mzcompose --find skip-version-upgrade list-workflows | grep -q "default"
BrokenPipeError: [Errno 32] Broken pipe

Adding a small sleep between each print leads to a reliable failure:

diff --git a/misc/python/materialize/cli/mzcompose.py b/misc/python/materialize/cli/mzcompose.py
index 3a2f5dbb81..725150ed00 100644
--- a/misc/python/materialize/cli/mzcompose.py
+++ b/misc/python/materialize/cli/mzcompose.py
@@ -319,6 +319,7 @@ class ListWorkflowsCommand(Command):
         composition = load_composition(args)
         for name in sorted(composition.workflows):
             print(name)
+            time.sleep(0.1)


 class DescribeCommand(Command):

Without PYTHONUNBUFFERED it would have taken us 8 KiB of output before we’d run into the broken pipe, none of our tests has that many workflows.

That concludes this long investigation into what turned out to be an interesting flake that we’ve been seeing in CI! The trivial fix is removing the -q from the grep call, but we could also have disabled pipefail for that one pipe or we could have unset PYTHONUNBUFFERED. One lesson from this debugging session is that even simple behaviors can combine in unexpected ways to create strange flaky failures. In this case all of these conspired to make this CI flake possible:

Misdirection: Run different code in CI compared to locally
Switch to another kind of server with slightly different performance characteristics
Disable Python’s output buffering
Enable Bash’s pipefail
Make grep run quietly

In hindsight I realized why this specific skip-version-upgrade/mzcompose.py was having the flake most often. It is one of the smallest test files:

$ wc -l test/*/mzcompose.py | sort -n
      39 test/pg-rtr/mzcompose.py
      43 test/mysql-rtr/mzcompose.py
      67 test/debezium/mzcompose.py
[...]
      85 test/skip-version-upgrade/mzcompose.py
[...]
     966 test/0dt/mzcompose.py
    1333 test/bounded-memory/mzcompose.py
    1916 test/limits/mzcompose.py
    4767 test/cluster/mzcompose.py

But other than the test definitions that are even smaller and never had this no such service: default error, the skip-version-upgrade/mzcompose.py file contains multiple workflows, so the smaller files could not run into the issue at all:

$ grep "def workflow_" test/*/mzcompose.py | cut -d":" -f1 | \
  uniq -c | sort -n | grep -v "   1 "
   2 test/skip-version-upgrade/mzcompose.py
   3 test/canary-environment/mzcompose.py
[...]
  57 test/cluster/mzcompose.py

Testing Materialize

2024-05-14T00:00:00-04:00

I joined Materialize’s Quality Assurance (QA) development team over a year ago. Since our team is small, we have to be conscious about focusing our time on the most impactful testing and test tooling.

Our goal is to find issues in Materialize as early and as efficiently as possible, ideally before a pull request even lands. Despite our small team, Materialize has a huge surface area.

Read the rest of the blog post over on the Materialize blog.

Haskell: Game Programming with GIF Streams

2023-09-07T00:00:00-04:00

Note: This is translated from the German original, which was used as a homework for the Programming Paradigms course at Karlsruhe Institute of Technology a long long time ago when I was co-holding the practical courses.

Snake is a computer game in which a snake has to be moved through a playing field. Eating food increases the snake’s length. When the snake collides with a wall or itself the game ends.

In this homework you will implement Snake in Haskell. For this you will require the framework from the gifstream repository.

The output of the game happens in an animated GIF stream, which you can watch in your browser. 64 colors are supported, which can be respresented as Int tuples of (0,0,0) up to (3,3,3).

type RGB = (Int,Int,Int)

A single frame of a GIF is defined as a list of rows, wherein each row is a list of RGB values.

type Frame = [[RGB]]

The framework provides a \texttt{server} function, which runs an HTTP server on the supplied port. The server sends each client a new frame of the GIF animation at the set interval. In the passed logic function new frames will be generated dynamically.

server :: PortNumber -> Int -> Logic -> IO ()

The Snake.hs file contains the basis for writing the Snake game. Compile the game and run it (You’ll need to install the network and random Haskell packages too):

$ ghc -O3 -threaded Snake.hs
[1 of 2] Compiling GifStream        ( GifStream.hs, GifStream.o )
[2 of 2] Compiling Main             ( Snake.hs, Snake.o )
Linking Snake ...
$ ./Snake
Listening on http://127.0.0.1:5002/

Open the supplied address in a browser. By pressing the WASD keys in the terminal you can influence the GIF in your browser.

Other participants in your network can watch the GIF stream as well, by using your network IP address instead of 127.0.0.1.

Furthermore it is possible to record the GIF stream and watch it later:

wget -O game.gif http://127.0.0.1:5002/

The most important function in Snake.hs is logic:

logic wait getInput sendFrame = initialState >>= go
  where
    go (State oldAction snake food) = do
      input <- getInput

      -- Generate new state
      let action = charToAction input oldAction
      let newSnake = snake
      let newFood = food

      let frame = case action of
            MoveUp    -> replicate height (replicate width (3,0,0))
            MoveDown  -> replicate height (replicate width (0,3,0))
            MoveLeft  -> replicate height (replicate width (0,0,3))
            MoveRight -> replicate height (replicate width (3,3,3))

      sendFrame (scale zoom frame)

      wait
      go (State action newSnake newFood)

The logic function creates an initial state for the Snake game and passes it to the go function. This function in turn uses getInput to read the key which was pressed last. Afterwards a new game state is generated. The shown frame is chosen based on the pressed key. Finally sendFrame send the new frame to all connected clients. As part of this function each frame is scaled by the scale. The call to wait causes waiting for the set time delay, which defaults to 100 ms. At the end of the function it calls itself tail recursively with the newly generated state. The goal of this task is to extend the game logic in logic step by step so that you can play Snake in the end.

Printing the Playing Field

Use the current state to generate an image and print this instead of the simple single-color image.

Write a list fieldPositions which saves the coordinates of the playing field in its corresponding position.

fieldPositions :: [[Position]]

The size of the field is saved in width and height. For a field of the size 3x4 fieldPositions would look as follows:

fieldPositions = [[(0,0),(1,0),(2,0)]
                 ,[(0,1),(1,1),(2,1)]
                 ,[(0,2),(1,2),(2,2)]
                 ,[(0,3),(1,3),(2,3)]]

Implement a colorize function which maps a single position of the image to a color so that the new frame can be created through let frame = map (map (colorize newSnake newFood)) fieldPositions. A field should be colored differently depending on whether this position is part of the snake, food or background.

colorize :: [Position] -> Position -> Position -> RGB

Snake Behavior

Next implement state changes so that the game logic can be written as let newSnake = moveSnake snake food action.

moveSnake :: [Position] -> Position -> Action -> Position

A snake is defined as a list of positions. The new snake receives a new head depending on the passed action. Action is defined as follows:

data Action = MoveLeft | MoveRight | MoveUp | MoveDown deriving Eq

At the tail the last element is cut off, except when the snake just reached food.

It has to be ensured that the user-chosen action is even possible. Write a function validateAction so that the game logic can be extended by let action = validateAction oldAction (charToAction input oldAction). For this validateAction shall only return a new action when this is possible. Otherwise the old action shall be returned.

Picture from Learn You A Haskell

Food Behavior

Next implement the state change of the food so that the game logic can be extended by newFood <- moveFood newSnake food.

moveFood :: [Position] -> Position -> IO Position

When the snake doesn’t currently eat the food the old position of the food can be returned directly. Otherwise the new position of the food shall be chosen. Avoid that the food appears inside of the body of the snake

Random numbers between x and y (inclusively) can be generated in the do syntax using r <- randomRIO (x,y). Thus import the System.Random module.

End of the Game

Adapt the end of logic so that with newSnake the validity of the new state is checked. In an invalid state the game shall be restarted by calling initialstate >>= go.

checkGameOver :: [Position] -> Bool

A full solution is available in the GitHub repository.

Bonus

Program another game with GIF stream output, for example Pong, Tetris or Conway’s Game of Life.

Related: time.gif

God writes Haskell

2023-06-03T00:00:00-04:00

God famously does not play dice with the universe, but he seems to enjoy writing Haskell:

Consider the wave-particle duality in quantum mechanics. Every particle behaves as a wave, as long as you haven’t interacted with it. Thanks to Haskell’s lazy evaluation values are also only evaluated once they are accessed (interacted with particles), and stay unevaluated thunks (waves) in the meantime.
Two particles can be Quantum-entangled, so that their states depend on each other, even though the particles are seperated by any distance. In Haskell a value, whether it’s evaluated yet or not, can also be shared and then used in a totally different location in the program without having to copy it. The value is even immutable, so that you can’t change it from one location and thus influence the other. Similarly for entangled particles you can’t manipulate one to change the state of the other particle, which might be far away and thus break the maximum speed of information.
Since values are immutable they have to be cleaned up more often in Haskell than typically in imperative languages. GHC, the most commonly used Haskell compiler, allocates new data in a special area. Only after a supernova will the still-relevant data be ejected into the larger universe.
Haskell beginners often use lists instead of arrays. You can’t do random access in a linked list, but only access the first element and then the rest of the list. The real world also doesn’t allow you random access, you are limited by the speed of light and have to go from one location to the next.
Time also seems to be a linked list, not even doubly linked, since you can’t go back after accessing the current element. Seems like an awkward bug.
Since the Haskell type system is so good at catching bugs, you often feel like you don’t even need to write tests. This is unfortunately untrue, as the strange physical bugs of our universe demonstrate: The speed of light happens to stay the same, no matter what speed you move at.
There seems to be a long-term memory leak in Haskell, which is kind of easy to happen since you can have unevaluated thunks growing in size as long as they are still reachable. Unfortunately this memory leak will keep growing until it consumes the entire universe, which will then die of its heat death.

Discuss on Hacker News

Database Testing at Yugabyte

2023-01-14T00:00:00-05:00

YugabyteDB is a cloud-native database for business-critical enterprise applications. It is designed to provide continuous availability as well as horizontal scalability, while retaining a strong set of RDBMS features. This objective creates a strong quality incentive for us in the Yugabyte Quality Assurance (QA) team. As a member of this team, I am giving an overview of the testing philosophy, approaches, and implementations for YugabyteDB.

Read the rest of the blog post over on the YugabyteDB blog.

macOS Setup after 15 Years of Linux

2021-12-26T00:00:00-05:00

About 3 years ago I wrote about my Linux Desktop Setup and having used pretty much the same software setup for 10-15 years. My setup used to be heavily keyboard based. Now for work I am for the first time using a macOS-based Macbook Pro with an M1 Pro CPU. I spent the last 3 weeks using it as a daily driver, tweaking it here and there. Here are my experiences.

Hardware-wise the machine is great. With the 10 core ARM64 CPU I get about 2-3 times the CPU performance of my desktop system with an Intel i7 6700k. In DDNet I can reach 1800 FPS which is clearly more than enough and about 10 times the FPS I have with the 6700k’s iGPU. I haven’t heard the fan turn on at all yet, so either my workloads, and even my compilations, are too light-weight, or the fan is pretty quiet. The integrated 120 Hz screen is far better than anything I’ve had before, with clearer colors and smoother motion.

Installing Software

macOS’s AppStore doesn’t seem to have most of the relevant applications for me. Homebrew seems to be the most popular third-party package manager for macOS, so I chose that, hoping that it would support most programs I want to use. Command line applications can be installed with brew install, while graphical applications that you want acccessible in the regular Applications directory need a brew install --cask.

For some applications like mpv this can be a bit confusing since there is both a command-line Formulae and a GUI Cask available for it. The Cask doesn’t support arm64 natively yet, since the maintainer of it doesn’t have an M1 Mac and mpv doesn’t provide official binaries. So here I noticed a major difference in how Homebrew seems to work compared to Arch Linux for example. On Arch Linux applications are built from source code centrally instead of relying on binaries from a vendor.

Luckily it was easy enough to build a DMG of mpv myself, which I then installed by dragging it into the Applications folder:

git clone https://github.com/mpv-player/mpv
cd mpv
git clean -f -d
git pull origin master
./bootstrap.py
export PKG_CONFIG_PATH="$(brew --prefix luajit-openresty)/lib/pkgconfig"
./waf configure --lua=luajit
./waf build
TOOLS/osxbundle.py -s build/mpv

With the openwith utility I can set mpv as the default application for video file types:

openwith io.mpv mkv mov mp4 avi

Compared to the x86-64 binary executed with the Rosetta2 translator this uses half the CPU. Surprisingly playing YouTube videos on Firefox directly is using even less CPU. My current assumption is that the videotoolbox hardware decoding support doesn’t support the same codecs in mpv yet as it does in Firefox.

Update: By building my own FFmpeg version VP9 videotoolbox support is available, it will probably be in the next FFmpeg release.

Command line software

My regular command line software like vim, mutt, remind, htop, unison, yt-dlp, newsboat, ssh, rsync, rrb work just fine and were easy to install with a brew install command. So most of my setup can stay the same and I can still have the same setup whether I run my applications in a local terminal or on my home server via SSH, for example from my phone or a remote machine.

Window management

I started missing xmonad, my tiling window manager of choice immediately. So my first idea was to use Amethyst instead, which provides some of the features of xmonad, but without the customizability of writing your own config in Haskell.

It seemed to work ok, but was feeling a bit more awkward to use compared to on Linux. So I decided to use the native macOS window management instead and try to get used to it instead of actively fighting against it. At the moment I’m using the integrated 16” screen and an external 24” monitor. Most applications I use “zoomed in”, which means they take all the space, but are not actually fullscreen, so you still see the dock and window’s title bar. There appears to be no system shortcut to zoom a window, so I have to double click the title bar.

Update: I found out that you can set a shortcut for “Zoom” by adding it in the System Preferences. This apparently works for any term that is available in the program’s title bar:

Update 2: Several people suggested Rectangle for moving and resizing windows, which seems to work well, especially with my large 40” screen.

Terminal

iTerm2 is the terminal emulator I chose to use. It has lots of features and so far seems to work fine. Each window and tab gets a dedicated shortcut, and it allows splitting the tab horizontally and vertically, so I can have multiple terminals open at once and jump between them reasonably without needing to use the mouse.

I mostly have one iTerm window open for most command line applications. If I need something quickly, I bound command-return to open a hotkey terminal window. Opening a new iTerm window from outside the application was slightly more challenging. The easiest way is pressing command-n from inside iTerm. From outside just using F4 and writing iTerm will do nothing when another iTerm window is already open. I had to set up an AppleScript workflow in Automator:

on run {input, parameters}
    tell application "iTerm"
        create window with default profile
    end tell
    return input
end run

Then it can be set as a global shortcut in Preferences:

There seems to be considerable overhead to using Automator for shortcuts though, the terminal window takes nearly a second to appear. Luckily I rarely needed a new terminal, since the windows persist after a reboot and I mostly just have to set them up once, and the hotkey terminal is fine for quick tasks like checking mail.

One issue I still have is that iTerm seems to send the home/end/insert keys differently from urxvt, my terminal emulator on Linux. But since both are identifying as TERM=xterm-256color to support full colors, I can’t tell them apart using zkbd when SSHing into a remote system. Not sure yet how to solve this cleanly. Here are the differences:

My favorite fixed-width bitmap font Terminus works fine with anti-aliasing disabled in iTerm with the font-terminus formula installed via:

brew tap homebrew/cask-fonts
brew install font-terminus

Unfortunately the font only works at specific sizes, I haven’t figured out yet how to exclude them when changing the font-sizes on-the-fly using command-+ and command–. But I rarely need to change font sizes anyway.

I still use my tach script so I don’t use my terminals when I accidentally close the iTerm windows:

tach is a script that provides automatic detaching and reattaching of terminals using dtach

This also allows me to move a running terminal session from one place to another using only the keyboard by detaching it using command-w and opening a new terminal anywhere I want, which will contain the previously detached session.

Keyboard layout

I’ve been using a German keyboard layout for a while, since a short excursion to Dvorak. My first problem was that macOS’s German keyboard layout uses option-l instead of the option-q I was used to for typing the @ character. Luckily this is easily remedied by switching from the “German” keyboard layout to “German - Standard”.

Unfortunately this is still not quite the same as on a German standard PC keyboard, where the altgr key sits just right of the spacebar, while on a Mac the option key is seperated from the spacebar by the command key. The problem is exacerbated by command-q being the default keyboard shortcut to close an application on macOS. So I ended up closing my currently open application every time I tried entering an @, for example when writing my own email address trying to log into a website.

That’s not ideal, so I thought the easiest solution would be switching the command and option modifier keys in the System Preferences:

This brought me one step forward, two steps back: Now the default system key binds are more akward, since this also switches the left option and command keys. Apple uses command-f (search), command-c (copy), command-v (paste) etc. and all of these became a bit more awkward to press with the preferences-based modifier key switch.

Additionally I noticed that the left option key was actually switched with the right command key and the left command key with the right option key, which is also not ideal when you have programs that only listen to one of the modifiers. At least I got my first macOS bug report out of this.

A better solution for me was using Karabiner-Elements to only swap the right command and option keys:

Of course I also remapped capslock as a secondary escape key, which is commonly required in Vim to leave insert mode. This also worked fine in the System Preferences for what it’s worth.

It’s unfortunate this modifier remapping has to be done with an external application, which can now read all my keyboard inputs system-wide, and thus has to be trusted.

Update: Thanks to Anriudh for suggesting an easier solution for the remapping. This file in ~/Library/LaunchAgents/com.local.KeyRemapping.plist takes care of the problem, no more Karabiner-Elements required:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.local.KeyRemapping</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/hidutil</string>
        <string>property</string>
        <string>--set</string>
        <string>{"UserKeyMapping":[

            {
              "HIDKeyboardModifierMappingSrc": 0x7000000E7,
              "HIDKeyboardModifierMappingDst": 0x7000000E6
            },

            {
              "HIDKeyboardModifierMappingSrc": 0x7000000E6,
              "HIDKeyboardModifierMappingDst": 0x7000000E7
            },

            {
              "HIDKeyboardModifierMappingSrc": 0x700000039,
              "HIDKeyboardModifierMappingDst": 0x700000029
            }

        ]}</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Deadkeys

On Linux I’ve always been using a nodeadkeys keyboard layout, which means keys like ^ and ´ get output immediately instead of being combined with the next letter entered. There seems to be no support for this on macOS, so I had to make my own keyboard layout using Ukelele:

For remapping modifier keys Ukelele recommends still using Karabiner, so unfortunately we can’t solve all keyboard layout problems with a single new keyboard layout:

Warning!

Ukelele can change the output of special keys, but this doesn’t always work as expected, or for all applications. There are usually other ways to achieve what you want.

• If you want to rearrange the modifier keys, try System Preferences.

• If you want to make other modifications such as turning modifier keys into normal keys, or normal keys into modifier keys, Karabiner (https://pqrs.org/osx/karabiner/) can likely help you.

• If you would like to be able to use the F-keys (F1 through F19), then Better Touch Tool (https://www.boastr.net/) can probably help you.

This warning appears once per session, and does not stop you changing special key output.

Ukelele also allowed me to change command-spacebar from outputting a non breaking space (aka   in HTML) to a regular space. It was easy to still have the option key pressed after writing a symbol on the German keyboard layout, so the following space often became a non breaking space for me, which threw off code linters.

Finally the created keyboard layout bundle file can be copied into /Library/Keyboard Layouts/ and then be selected in the system settings. It seems not to work during the Login screen, but I don’t mind much.

Mouse

I’m using a Logitech MX Master 3 connected via Bluetooth. I want it to scroll normally (up means up, down means down). But I want to keep the “natural” scrolling direction of the touch pad, which works like scrolling on a smart phone, so you move two fingers up to scroll down, two fingers down to scroll up. With the system-wide settings it seems only possible to change this for both the touchpad and the mouse at once.

Logitech Options luckily supports changing this just for the mouse, but always takes a few seconds at startup to register the device and start working.

Unfortunately the mouse scrolling seems to have an acceleration, which I have not been able to disable yet.

Update: Luc sent me this command which seems to improve the acceleration situation, but it still doesn’t feel as linear as on Linux:

defaults write .GlobalPreferences com.apple.scrollwheel.scaling -1

Update: Thanks to Kristjan for suggesting LinearMouse, which solves the acceleration problem for me. No more Logitech Options required either to reverse scrolling wheel on mouse only.

Update 2: LinearMouse is somehow limiting my mouse wheel speed when scrolling very quickly, so I still prefer Logitech Options.

Putting the display to sleep easily by moving the mouse cursor in the corner is nice (I’d still prefer a keyboard shortcut), and can be enabled in the system preferences:

Task bar

I use Stats to display some network and CPU statistics in the task bar, works solidly so far and has replaced conky for me.

Firefox & Web

Safari doesn’t support uBlock Origin, see this explanation by ghost:

Very quick tl;dr: uBO will no longer work with Safari, use Firefox or a new “content blocker” app (see below for good replacements).

So this made the choice easy for me, since Chrome will also phase out uBlock Origin support. I simply kept using Firefox with my essential addons: uBlock Origin and my password manager Bitwarden.

I figured out that it’s quite easy to have my command line email client mutt preview HTML emails inline, but still open Firefox by pressing m on the HTML part, by adding this to my .mailcap file:

text/html; /Applications/Firefox.app/Contents/MacOS/firefox '%s' &; needsterminal
text/html; links -dump %s; needsterminal; copiousoutput;

To handle URLs and other texts copied on the command line automatically, pbpaste/pbcopy replace xclip as the clipboard access tools on macOS. Downloading a video from the URL in clipboard for example:

pbpaste|xargs yt-dlp --

Music

I tried using the integrated Music application, but it kept struggling to import my music database. This might have been slow since I connected to it using macFUSE & SSHFS. I don’t need a fancy music player anyway and remembered that > 15 years ago I used to use Foobar2000 on Windows. It’s still around and there is a macOS version, so I’m just using that for now.

With a longer read-ahead I haven’t had any pauses in music playback so far, even though the sshfs connection is over the internet. Sound quality of the MacBook is fine, I haven’t noticed any problems and the dedicated headphone jack is appreciated (as is the half-size SD card slot for loading music onto the SD card for my car’s stereo).

The audio volume controls are a bit too coarse-grained for me, but using shift-option-F11/F12 allows for a few more steps when setting the volume.

Translations

I used to use rdictcc to easily access the excellent German-English translations from wiki-like dictionary service dict.cc. There is a dict.cc plugin for the macOS dictionary which I am now using instead.

On macOS translations are easily accessible by holding the mouse over a word or a selected term and pressing control-command-d.

Standby

To get system standby not to wake up constantly I had to disable “Wake for network access” in Battery settings: I also previously set

defaults write /Library/Preferences/com.apple.mDNSResponder.plist NoMulticastAdvertisements -bool YES

since the DNS responder kept waking up the system according to pmset -g log|grep "Wake Requests", but this might also be fixed by the “Wake for network access” setting.

Useful shortcuts

Here are some of the default system shortcuts I’ve been using to get over not using xmonad anymore as my window manager (and Vimperator/Pentadactyl for Firefox respectively, but that switch I already had to make a few years ago):

Command	Application	Shortcut
Open app		🌐 F4
Next app		⌘ ⇥
Previous app		⇧ ⌘ ⇥
Next desktop		⌃ ►
Previous desktop		⌃ ◄
Mission control		⌃ ▲ / 🌐 F3
Move app to next desktop		Click and hold title bar, ⌃ ►
Move app to previous desktop		Click and hold title bar, ⌃ ◄
New window		⌘ N
Close window		⌘ Q
Move focus to next window		⌘ `
New tab	iTerm, Firefox	⌘ T
Close tab	iTerm, Firefox	⌘ W
Next tab	iTerm, Firefox	⌃ ⇥
Previous tab	iTerm, Firefox	⇧ ⌃ ⇥
Restore tab	Firefox	⇧ ⌘ T
Split vertically	iTerm	⌘ D
Split horizontally	iTerm	⇧ ⌘ D
Select upper pane	iTerm	⌥ ⌘ ▲
Select lower pane	iTerm	⌥ ⌘ ▼
Select right pane	iTerm	⌥ ⌘ ►
Select left pane	iTerm	⌥ ⌘ ◄
Increase font size	iTerm, Firefox	⌘ +
Decrease font size	iTerm, Firefox	⌘ -
Search		⌘ F
Copy		⌘ C
Cut		⌘ X
Paste		⌘ V
Save		⌘ S
Select all		⌘ A
Select URL field	Firefox	⌘ L
Open URL	iTerm	⌘ Left mouse click
Dictionary lookup		⌃ ⌘ D
Take screenshot of area		⇧ ⌘ 4
Take screenshot of window		⇧ ⌘ 4, ⎵
Increase volume slightly		⇧ ⌥ F12
Decrease volume slightly		⇧ ⌥ F11

And since some keys are missing on the MacBook keyboard here is how to get them:

Key	Shortcut
Page up	🌐 ▲
Page down	🌐 ▼
Home	🌐 ◄
End	🌐 ►
Delete	🌐 ⌫
Insert	🌐 ⏎

Conclusion

macOS is working better than expected for me so far, and definitely better than Windows would. So I’m glad with my choice of a MacBook Pro as a work device. Now that I’ve written down some of the steps I had to take to make it more usable, I hopefully won’t forget them if I ever have to set a macOS based system up again. If you have any questions or tips get in touch with me at dennis@felsing.org.

Discussion on Hacker News.

Quest for the Lost Home Server

2019-10-15T00:00:00-04:00

Today I lost access to my home server. As I described in a previous post I depend heavily on the server to fetch my emails, as a file server, to synchronize files, for newsbeuter and irssi sessions and many other things. As no one was going to be in proximity of the server for the next few hours, my goal for today was to solve the problem remotely.

The symptom was that my SSH connection attempts failed. The server also didn’t respond to pings. As the server is sitting at home behind a regular DSL connection it uses a dynamic IP address that is shuffled every 24 hours. So my first hunch was that the daily reconnect might just have happened at a different time today and I gave the server some time to broadcast its new IP address to my domain registrar.

After about an hour I still couldn’t connect, so I started investigating. Maybe the API of the domain registrar changed (as happened before) or my script failed for another reason? I know that my home server does nightly backups of the other servers I run. So I connected to one of them and checked the journalctl log. To my surprise no connection from the server happened last night. My worst fear was that the server was hanging due to a hardware problem, as I ran into similar problems with this Intel Bay Trail CPU before. (The issue seems to be that Intel underdesigned the power delivery on those systems, which the graphics driver is trying to work around.)

But I wasn’t ready to give up yet, so I tried to think of any other activities the server would do that might leave a trace. I found out that my email hoster doesn’t provide easy access to the IP addresses that access the mailbox. I couldn’t think of any other traces at the time, but next time I might check if my IP address is visible on some IRC server.

As a last resort I remembered that my ISP usually gives me quite similar IP addresses, so I used whois on yesterday’s IP address to see how large the subnet is. I got back a subnet of only 2¹⁴ addresses, which seemed reasonable enough to scan. I hope the ISP is not too mad at me for such a small port scan, but to reduce the impact I also told nmap to only scan the specific SSH port that I use.

After about 5 minutes I had found a single server responding on my custom SSH port, which was indeed my home server. After logging in and checking out the system I noticed the (usual) problem of cronie being unable to spawn new processes after a glibc update. So I simply restarted cronie to fix the problem.

I still wish that Arch Linux will one day tell me which running executables to restart after a system upgrade, as SLES does, to prevent problems like this in the future. I think the general approach is just to check which process references deleted shared libraries. Something like this, similar to here:

ps axh -o pid | while read PID; do grep ".so" /proc/$PID/maps | grep "(deleted)" && echo "$PID" && sed -e "s/\x00/ /g" < /proc/$PID/cmdline && echo "\n"; done

Update: overdue is a pacman post-transaction hook that lists daemons that still reference old shared libraries.

Chinese Traffic to time.gif

2019-05-21T00:00:00-04:00

Nearly two years ago I posted this endless GIF that always shows the current time in UTC:

Now looking at my GoAccess dashboard I can see that it is picking up in popularity rather suddenly:

But strangely I can’t find anything about time.gif being linked on the web. So this might just be an attempted Denial of Service (DoS) attack? At least that would be something I am familiar with from the DDNet direction, but it’s certainly strange on HookRace. But instead of simply shutting down time.gif I decided to try to find out who is accessing it and whether I can keep the server up.

Let’s look into the nginx logs, since I use nginx to proxy the requests to the Haskell program. There I see about 40 new requests per second looking like this:

hookrace.net 123.185.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 3335 "XXX" "Mozilla/5.0 (Linux; Android 8.1.0; V1818A Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044681 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 8.055
hookrace.net 111.62.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 32061 "XXX" "Mozilla/5.0 (Linux; Android 5.1; OPPO A59s Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044704 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043B) Process/tools NetType/WIFI Language/zh_CN" 89.256
hookrace.net 111.29.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 543830 "XXX" "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044704 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 1540.238
hookrace.net 112.2.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 172102 "XXX" "Mozilla/5.0 (Linux; Android 8.1.0; V1816A Build/OPM1.171019.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044611 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 492.600
hookrace.net 123.13.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 1275 "XXX" "Mozilla/5.0 (Linux; Android 9; LYA-AL00 Build/HUAWEILYA-AL00L; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044704 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 1.888
hookrace.net 117.91.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 4684 "XXX" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/16D57 MicroMessenger/7.0.3(0x17000321) NetType/WIFI Language/zh_CN" 12.123

I checked a few IP addresses and they were all in mobile networks, not data centers. The user agent containing MicroMessenger and MQQBrowser indicates that the source of the traffic are WeChat and/or QQ, popular chinese chat apps.

Quantifying the Traffic

For reference, the system I’m running on is a simple Debian based VPS with 2 threads and 2 GB of RAM that also functions as the main server for DDNet’s website, database and my HookRace blog.

I already had to do some scaling when posting the initial blog post on Hacker News, optimizing the Haskell application itself to use LZW encoding in the GIF frames, to properly clean up connections to prevent any memory leaks and disable buffering in nginx’s config.

But the current level of traffic is on a different scale with 2.4 million hits on time.gif in the last 23 hours (30 hits per second) resulting in 113 GB of data being transferred. And many of those connections don’t finish quickly, instead they linger for seconds, minutes or even hours.

Using lsof -i | grep Time | wc -l I can see that there are about 6000 people downloading the GIF at peak times, causing up to 30 Mbit/s of outgoing traffic with 7000 packets/second incoming and the same number outgoing. The DDNet server statistics lets me monitor this nicely (related blog article):

Network Packets CPU

Keeping Up with the Traffic

Regenerating the ranks pages of DDNet usually causes the main CPU load on the server, which can be seen in the above CPU graph as spikes. This task is already set to only run when the server is below a specified load, so that more essential tasks have priority.

The first new problem was nginx running into a limit of 768 worker_connections:

2019/05/20 20:41:30 [alert] 761#761: *3828093 768 worker_connections are not enough while connecting to upstream, client: 49.114.XXX.XXX, server: hookrace.net, request: "GET /time.gif HTTP/2.0", upstream: "http://127.0.0.1:5002/", host: "hookrace.net", referrer: "XXX"

Luckily that is easily fixed in /etc/nginx/nginx.conf by increasing the number of worker_connections to keep alive, each of which is handling one of the long-lasting time.gif requests:

events {
  worker_connections 20000;
}

and systemctl reload nginx. No downtime required since nginx will start new worker processes to handle new requests while keeping the old ones alive for a time to keep handling existing connections.

Unfortunately that fix only lasted a few hours until the next problem appeared:

2019/05/20 23:09:21 [alert] 15188#15188: *4041619 socket() failed (24: Too many open files) while connecting to upstream, client: 27.207.XXX.XXX, server: hookrace.net, request: "GET /time.gif HTTP/2.0", upstream: "http://127.0.0.1:5002/", host: "hookrace.net", referrer: "XXX"

Increasing the limits in /etc/security/limits.conf for the nginx user fixes this:

#<domain>      <type>  <item>         <value>
nginx          soft    nofile         1048576
nginx          hard    nofile         1048576

The value of 1048576 is chosen since it’s the value set in sysctl fs.file-max and it should be good enough for now.

Next I noticed that the server was running out of memory with both the Haskell application and nginx having to keep track of so many connections at once. For now I increased the swap size on the fly to keep some less commonly used stuff there using dd if=/dev/zero of=/var/swap bs=1M count=5000 && mkswap /var/swap && swapon /var/swap.

When running out of memory I noticed that Python’s msgpack implementation fails quite confusingly when it runs OOM. So I had to add some fixes to the code creating the DDNet ranks pages to handle this possibility.

The Linux Kernel’s TCP buffers ran out of memory next, complaining in dmesg:

[1638211.984805] TCP: out of memory -- consider tuning tcp_mem

So I increased them with a net.ipv4.tcp_mem = 116730 155640 233460 in /etc/sysctl.conf and reloaded it with sysctl -p.

A limitation of my current approach is the number of ports nginx can open to proxy to the Haskell application. If that gets blown I’ll have to communicate to the application differently or simply redirect to the application directly instead of proxying it. That would also reduce the CPU load significantly, cutting out nginx which happens to be much more expensive than the Haskell application, probably because it’s also handling TLS.

Final Words

While it was fun to keep time.gif running in the face of this amount of traffic, I still haven’t answered the final question of where this traffic is coming from. It might be that lots of Chinese happen to be spreading time.gif on WeChat and QQ, but for that the traffic looks a bit too sterile. Has anyone seen similar traffic patterns and might know if they are real or some kind of botnet? Maybe someone has embedded traffic.gif on some WeChat-specific page. If anyone has a clue please drop me an email at dennis@felsing.org.

The best guess so far is by Kolen:

Hi,

I read your post and this is just my guess:

Chinese “WhatsApp” type of communication culture is very strange. They are like spam emails in the old days. Often time some one might create posts in picture, eg include warm message such as reminding each other to put on more clothes as the weather is getting cold, etc.

My guess is then someone read your trick and thought that it is a good idea to create some picture that always show the current time. Eg to remind others that it is time to sleep or something.

And like email spams in the old days, people can share things like this crazily, often by older people who don’t know much about technology. (Just like how some tweet can go viral, messages like this could also go viral within there networks. And unfortunately the viral thing we often received are really rubbish like.)

yours,

Kolen

Discuss on Hacker News.

Linux Desktop Setup

2019-01-15T00:00:00-05:00

Russian Translation by Akhmad Karimov

My software setup has been surprisingly constant over the last decade, after a few years of experimentation since I initially switched to Linux in 2006. It might be interesting to look back in another 10 years and see what changed. A quick overview of what’s running as I’m writing this post:

Motivation

My software priorities are, in no specific order:

Programs should run on my local system so that I’m in control of them, this excludes cloud solutions.
Programs should run in the terminal, so that they can be used consistently from anywhere, including weak computers or a phone.
Keyboard focused is nearly automatic by using terminal software. I prefer to use the mouse where it makes sense only, reaching for the mouse all the time during typing feels like a waste of time. Occasionally it took me an hour to notice that the mouse wasn’t even plugged in.
Ideally use fast and efficient software, I don’t like hearing the fan and feeling the room heat up. I can also keep running older hardware for much longer, my 10 year old Thinkpad x200s is still fine for all the software I use.
Be composable. I don’t want to do every step manually, instead automate more when it makes sense. This naturally favors the shell.

Operating Systems

I had a hard start with Linux 12 years ago by removing Windows, armed with just the Gentoo Linux installation CD and a printed manual to get a functioning Linux system. It took me a few days of compiling and tinkering, but in the end I felt like I had learnt a lot.

I haven’t looked back to Windows since then, but I switched to Arch Linux on my laptop after having the fan fail from the constant compilation stress. Later I also switched all my other computers and private servers to Arch Linux. As a rolling release distribution you get package upgrades all the time, but the most important breakages are nicely reported in the Arch Linux News.

One annoyance though is that Arch Linux removes the old kernel modules once you upgrade it. I usually notice that once I try plugging in a USB flash drive and the kernel fails to load the relevant module. Instead you’re supposed to reboot after each kernel upgrade. There are a few hacks around to get around the problem, but I haven’t been bothered enough to actually use them.

Similar problems happen with other programs, commonly Firefox, cron or Samba requiring a restart after an upgrade, but annoyingly not warning you that that’s the case. SUSE, which I use at work, nicely warns about such cases.

For the DDNet production servers I prefer Debian over Arch Linux, so that I have a lower chance of breakage on each upgrade. For my firewall and router I used OpenBSD for its clean system, documentation and great pf firewall, but right now I don’t have a need for a separate router anymore.

Window Manager

Since I started out with Gentoo I quickly noticed the huge compile time of KDE, which made it a no-go for me. I looked around for more minimal solutions, and used Openbox and Fluxbox initially. At some point I jumped on the tiling window manager train in order to be more keyboard-focused and picked up dwm and awesome close to their initial releases.

In the end I settled on xmonad thanks to its flexibility, extendability and being written and configured in pure Haskell, a great functional programming language. One example of this is that at home I run a single 40” 4K screen, but often split it up into four virtual screens, each displaying a workspace on which my windows are automatically arranged. Of course xmonad has a module for that.

dzen and conky function as a simple enough status bar for me. My entire conky config looks like this:

out_to_console yes
update_interval 1
total_run_times 0

TEXT
${downspeed eth0} ${upspeed eth0} | $cpu% ${loadavg 1} ${loadavg 2} ${loadavg 3} $mem/$memmax | ${time %F %T}

And gets piped straight into dzen2 with conky | dzen2 -fn '-xos4-terminus-medium-r-normal-*-12-*-*-*-*-*-*-*' -bg '#000000' -fg '#ffffff' -p -e '' -x 1000 -w 920 -xs 1 -ta r.

One important feature for me is to make the terminal emit a beep sound once a job is done. This is done simply by adding a \a character to the PR_TITLEBAR variable in zsh, which is shown whenever a job is done. Of course I disable the actual beep sound by blacklisting the pcspkr kernel module with echo "blacklist pcspkr" > /etc/modprobe.d/nobeep.conf. Instead the sound gets turned into an urgency by urxvt’s URxvt.urgentOnBell: true setting. Then xmonad has an urgency hook to capture this and I can automatically focus the currently urgent window with a key combination. In dzen I get the urgent windowspaces displayed with a nice and bright #ff0000.

The final result in all its glory on my Laptop:

I hear that i3 has become quite popular in the last years, but it requires more manual window alignment instead of specifying automated methods to do it.

I realize that there are also terminal multiplexers like tmux, but I still require a few graphical applications, so in the end I never used them productively.

Terminal Persistency

In order to keep terminals alive I use dtach, which is just the detach feature of screen. In order to make every terminal on my computer detachable I wrote a small wrapper script. This means that even if I had to restart my X server I could keep all my terminals running just fine, both local and remote.

Shell & Programming

Instead of bash I use zsh as my shell for its huge number of features.

As a terminal emulator I found urxvt to be simple enough, support Unicode and 256 colors and has great performance. Another great feature is being able to run the urxvt client and daemon separately, so that even a large number of terminals barely takes up any memory (except for the scrollback buffer).

There is only one font that looks absolutely clean and perfect to me: Terminus. Since it’s a bitmap font everything is pixel perfect and renders extremely fast and at low CPU usage. In order to switch fonts on-demand in each terminal with CTRL-WIN-[1-7] my ~/.Xdefaults contains:

URxvt.font: -xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*
dzen2.font: -xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*

URxvt.keysym.C-M-1: command:\033]50;-xos4-terminus-medium-r-normal-*-12-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-2: command:\033]50;-xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-3: command:\033]50;-xos4-terminus-medium-r-normal-*-18-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-4: command:\033]50;-xos4-terminus-medium-r-normal-*-22-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-5: command:\033]50;-xos4-terminus-medium-r-normal-*-24-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-6: command:\033]50;-xos4-terminus-medium-r-normal-*-28-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-7: command:\033]50;-xos4-terminus-medium-r-normal-*-32-*-*-*-*-*-*-*\007

URxvt.keysym.C-M-n: command:\033]10;#ffffff\007\033]11;#000000\007\033]12;#ffffff\007\033]706;#00ffff\007\033]707;#ffff00\007
URxvt.keysym.C-M-b: command:\033]10;#000000\007\033]11;#ffffff\007\033]12;#000000\007\033]706;#0000ff\007\033]707;#ff0000\007

For programming and writing I use Vim with syntax highlighting and ctags for indexing, as well as a few terminal windows with grep, sed and the other usual suspects for search and manipulation. This is probably not at the same level of comfort as an IDE, but allows me more automation.

One problem with Vim is that you get so used to its key mappings that you’ll want to use them everywhere.

Python and Nim do well as scripting languages where the shell is not powerful enough.

System Monitoring

htop works great for getting a quick overview of what the software is currently doing. lm_sensors allows monitoring the hardware temperatures, fans and voltages. powertop is a great little tool by Intel to find power savings. ncdu lets you analyze disk usage interactively.

nmap, iptraf-ng, tcpdump and Wireshark are essential tools for analyzing network problems.

There are of course many more great tools.

Mails & Synchronization

On my home server I have a fetchmail daemon running for each email acccount that I have. Fetchmail just retrieves the incoming emails and invokes procmail:

#!/bin/sh
for i in /home/deen/.fetchmail/*; do
  FETCHMAILHOME=$i /usr/bin/fetchmail -m 'procmail -d %T' -d 60
done

The configuration is as simple as it could be and waits for the server to inform us of fresh emails:

poll imap.1und1.de protocol imap timeout 120 user "dennis@felsing.org" password "XXX" folders INBOX keep ssl idle

My .procmailrc config contains a few rules to backup all mails and sort them into the correct directories, for example based on the mailing list id or from field in the mail header:

MAILDIR=/home/deen/shared/Maildir
LOGFILE=$HOME/.procmaillog
LOGABSTRACT=no
VERBOSE=off
FORMAIL=/usr/bin/formail
NL="
"

:0wc
* ! ? test -d /media/mailarchive/`date +%Y`
| mkdir -p /media/mailarchive/`date +%Y`

# Make backups of all mail received in format YYYY/YYYY-MM
:0c
/media/mailarchive/`date +%Y`/`date +%Y-%m`

:0
* ^From: .*(.*@.*.kit.edu|.*@.*.uka.de|.*@.*.uni-karlsruhe.de)
$MAILDIR/.uni/

:0
* ^list-Id:.*lists.kit.edu
$MAILDIR/.uni-ml/

[...]

To send emails I use msmtp, which is also great to configure:

account default
host smtp.1und1.de
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on
from dennis@felsing.org
user dennis@felsing.org
password XXX

[...]

But so far the emails are still on the server. My documents are all stored in a directory that I synchronize between all computers using Unison. Think of Unison as a bidirectional interactive rsync. My emails are part of this documents directory and thus they end up on my desktop computers.

This also means that while the emails reach my server immediately, I only fetch them on deman instead of getting instant notifications when an email comes in.

From there I read the mails with mutt, using the sidebar plugin to display my mail directories. The /etc/mailcap file is essential to display non-plaintext mails containing HTML, Word or PDF:

text/html;w3m -I %{charset} -T text/html; copiousoutput
application/msword; antiword %s; copiousoutput
application/pdf; pdftotext -layout /dev/stdin -; copiousoutput

News & Communication

Newsboat is a nice little RSS/Atom feed reader in the terminal. I have it running on the server in a tach session with about 150 feeds. Filtering feeds locally is also possible, for example:

ignore-article "https://forum.ddnet.org/feed.php" "title =~ \"Map Testing •\" or title =~ \"Old maps •\" or title =~ \"Map Bugs •\" or title =~ \"Archive •\" or title =~ \"Waiting for mapper •\" or title =~ \"Other mods •\" or title =~ \"Fixes •\""

I use Irssi the same way for communication via IRC.

Calendar

remind is a calendar that can be used from the command line. Setting new reminders is done by editing the rem files:

# One time events
REM 2019-01-20 +90 Flight to China %b

# Recurring Holidays
REM 1 May +90 Holiday "Tag der Arbeit" %b
REM [trigger(easterdate(year(today()))-2)] +90 Holiday "Karfreitag" %b

# Time Change
REM Nov Sunday 1 --7 +90 Time Change (03:00 -> 02:00) %b
REM Apr Sunday 1 --7 +90 Time Change (02:00 -> 03:00) %b

# Birthdays
FSET birthday(x) "'s " + ord(year(trigdate())-x) + " birthday is %b"
REM 16 Apr +90 MSG Andreas[birthday(1994)]

# Sun
SET $LatDeg 49
SET $LatMin 19
SET $LatSec 49
SET $LongDeg -8
SET $LongMin -40
SET $LongSec -24

MSG Sun from [sunrise(trigdate())] to [sunset(trigdate())]
[...]

Unfortunately there is no Chinese Lunar calendar function in remind yet, so Chinese holidays can’t be calculated easily.

I use two aliases for remind:

rem -m -b1 -q -g

to see a list of the next events in chronological order and

rem -m -b1 -q -cuc12 -w$(($(tput cols)+1)) | sed -e "s/\f//g" | less

to show a calendar fitting just the width of my terminal:

Dictionary

rdictcc is a little known dictionary tool that uses the excellent dictionary files from dict.cc and turns them into a local database:

$ rdictcc rasch
====================[ A => B ]====================
rasch:
    - apace
    - brisk [speedy]
    - cursory
    - in a timely manner
    - quick
    - quickly
    - rapid
    - rapidly
    - sharpish  [Br.]  [coll.]
    - speedily
    - speedy
    - swift
    - swiftly
rasch [gehen]:
    - smartly [quickly]
Rasch {n} [Zittergras-Segge]:
    - Alpine grass [Carex brizoides]
    - quaking grass sedge [Carex brizoides]
Rasch {m} [regional] [Putzrasch]:
    - scouring pad
====================[ B => A ]====================
Rasch model:
    - Rasch-Modell {n}

Writing and Reading

I have a simple todo file containing my tasks, that is basically always sitting open in a Vim session. For work I also use the todo file as a “done” file so that I can later check what tasks I finished on each day.

For writing documents, letters and presentations I use LaTeX for its superior typesetting. A simple letter in German format can be set like this for example:

\documentclass[paper = a4, fromalign = right]{scrlttr2}
\usepackage{german}
\usepackage{eurosym}
\usepackage[utf8]{inputenc}
\setlength{\parskip}{6pt}
\setlength{\parindent}{0pt}

\setkomavar{fromname}{Dennis Felsing}
\setkomavar{fromaddress}{Meine Str. 1\\69181 Leimen}
\setkomavar{subject}{Titel}

\setkomavar*{enclseparator}{Anlagen}

\makeatletter
\@setplength{refvpos}{89mm}
\makeatother

\begin{document}
\begin{letter} {Herr Soundso\\Deine Str. 2\\69121 Heidelberg}
\opening{Sehr geehrter Herr Soundso,}

Sie haben bei mir seit dem Bla Bla Bla.

Ich fordere Sie hiermit zu Bla Bla Bla auf.

\closing{Mit freundlichen Grüßen}

\end{letter}
\end{document}

Further example documents and presentations can be found over at my private site.

To read PDFs Zathura is fast, has Vim-like controls and even supports two different PDF backends: Poppler and MuPDF. Evince on the other hand is more full-featured for the cases where I encounter documents that Zathura doesn’t like.

Graphical Editing

GIMP and Inkscape are easy choices for photo editing and interactive vector graphics respectively.

In some cases Imagemagick is good enough though and can be used straight from the command line and thus automated to edit images. Similarly Graphviz and TikZ can be used to draw graphs and other diagrams.

Web Browsing

As a web browser I’ve always used Firefox for its extensibility and low resource usage compared to Chrome.

Unfortunately the Pentadactyl extension development stopped after Firefox switched to Chrome-style extensions entirely, so I don’t have satisfying Vim-like controls in my browser anymore.

Media Players

mpv with hardware decoding allows watching videos at 5% CPU load using the vo=gpu and hwdec=vaapi config settings. audio-channels=2 in mpv seems to give me clearer downmixing to my stereo speakers / headphones than what PulseAudio does by default. A great little feature is exiting with Shift-Q instead of just Q to save the playback location. When watching with someone with another native tongue you can use --secondary-sid= to show two subtitles at once, the primary at the bottom, the secondary at the top of the screen

My wirelss mouse can easily be made into a remote control with mpv with a small ~/.config/mpv/input.conf:

MOUSE_BTN5 run "mixer" "pcm" "-2"
MOUSE_BTN6 run "mixer" "pcm" "+2"
MOUSE_BTN1 cycle sub-visibility
MOUSE_BTN7 add chapter -1
MOUSE_BTN8 add chapter 1

youtube-dl works great for watching videos hosted online, best quality can be achieved with -f bestvideo+bestaudio/best --all-subs --embed-subs.

As a music player MOC hasn’t been actively developed for a while, but it’s still a simple player that plays every format conceivable, including the strangest Chiptune formats. In the AUR there is a patch adding PulseAudio support as well. Even with the CPU clocked down to 800 MHz MOC barely uses 1-2% of a single CPU core.

My music collection sits on my home server so that I can access it from anywhere. It is mounted using SSHFS and automount in the /etc/fstab/:

root@server:/media/media  /mnt/media  fuse.sshfs noauto,x-systemd.automount,idmap=user,IdentityFile=/root/.ssh/id_rsa,allow_other,reconnect 0 0

Cross-Platform Building

Linux is great to build packages for any major operating system except Linux itself! In the beginning I used QEMU to with an old Debian, Windows and Mac OS X VM to build for these platforms.

Nowadays I switched to using chroot for the old Debian distribution (for maximum Linux compatibility), MinGW to cross-compile for Windows and OSXCross to cross-compile for Mac OS X.

The script used to build DDNet as well as the instructions for updating library builds are based on this.

Backups

As usual, we nearly forgot about backups. Even if this is the last chapter, it should not be an afterthought.

I wrote rrb (reverse rsync backup) 10 years ago to wrap rsync so that I only need to give the backup server root SSH rights to the computers that it is backing up. Surprisingly rrb needed 0 changes in the last 10 years, even though I kept using it the entire time.

The backups are stored straight on the filesystem. Incremental backups are implemented using hard links (--link-dest). A simple config defines how long backups are kept, which defaults to:

KEEP_RULES=( \
   7  7 \ # One backup a day for the last 7 days
  31  8 \ # 8 more backups for the last month
 365 11 \ # 11 more backups for the last year
1825  4 \ # 4 more backups for the last 5 years
)

Since some of my computers don’t have a static IP / DNS entry and I still want to back them up using rrb I use a reverse SSH tunnel (as a systemd service) for them:

[Unit]
Description=Reverse SSH Tunnel
After=network.target

[Service]
ExecStart=/usr/bin/ssh -N -R 27276:localhost:22 -o "ExitOnForwardFailure yes" server
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

Now the server can reach the client through ssh -p 27276 localhost while the tunnel is running to perform the backup, or in .ssh/config format:

Host cr-remote
  HostName localhost
  Port 27276

While talking about SSH hacks, sometimes a server is not easily reachable thanks to some bad routing. In that case you can route the SSH connection through another server to get better routing, in this case going through the USA to reach my Chinese server which had not been reliably reachable from Germany for a few weeks:

Host chn.ddnet.org
  ProxyCommand ssh -q usa.ddnet.org nc -q0 chn.ddnet.org 22
  Port 22

Final Remarks

Thanks for reading my random collection of tools. I probably forgot many programs that I use so naturally every day that I don’t even think about them anymore. Let’s see how stable my software setup stays in the next years. If you have any questions, feel free to get in touch with me at dennis@felsing.org.

Comments on Hacker News.

Theme Switcher in Nginx

2019-01-09T00:00:00-05:00

Some people really liked the dark DDNet theme for Halloween by Soreu, so we decided to keep it possible to use the default bright or the dark theme.

Thanks to xse we got a JavaScript based theme switcher. After some improvements I finally I switched it away from JavaScript entirely and finally am also using it on this blog with an OLED friendly dark theme.

The final result is quite simple, and only requires static HTML, a cookie for the theme and Nginx for setting and returning the correct CSS file for it. You can try it out by clicking on the Switch Theme button at the top of this page, which just redirects you to /switch-theme/.

Doing the CSS decision server-side instead of in JavaScript has the advantage that you don’t get any flicker on rendering, no matter what theme you chose.

The entire nginx logic is:

location /switch-theme/ {
  if ($http_cookie ~* "user_theme=/public/css-dark.css") {
    add_header Set-Cookie "user_theme=/public/css-empty.css;path=/;domain=hookrace.net";
    return 302 $http_referer;
  }
  add_header Set-Cookie "user_theme=/public/css-dark.css;path=/;domain=hookrace.net";
  return 302 $http_referer;
}

location /public/css-dark.css {
  if ($http_cookie ~* "user_theme=/public/css-dark.css") {
    return 302 /public/css/css-dark.css;
  }
  return 302 /public/css/css-empty.css;
}

Note that the /switch-theme/ location sends the user back to the $http_referer. /public/css-dark.css is statically included on each page, but what it does depends on the cookie value.

The 302 Moved Temporarily response makes the client send another request, adding another roundtrip to the page rendering. If the CSS is small enough you can just pull it directly into the Nginx config:

location /public/css-dark.css {
  if ($http_cookie ~* "user_theme=/public/css-dark.css") {
    return 200 "body {color: #fff; background-color: #000;} .page-title, .post-title, .post-title a, h1, h2, h3, h4, h5, h6 {color: #ccc;} .masthead-title a {color: #bbb;}";
  }
  return 200 ""; 
}

One side effect of this entire approach is that the CSS file can’t be cached anymore though, otherwise we’d end up showing the old theme when it gets switched. I’m sure this could be optimized somehow to tell the browser to only fetch the CSS file again after the theme was switched. But for my site the current performance is good enough.

Also note that If Is Evil in Nginx, so if you abuse features like this you might run into trouble.

Turning Down a Blockchain Job Offer

2018-01-22T00:00:00-05:00

I have recently received a job offer to work on a blockchain implementation. While the offer was very generous, I had to turn it down. In this post I want to collect the thoughts that went into my decision process leading to this conclusion.

Introduction

I have not touched blockchain technology much before, other than reading the original Bitcoin paper shortly after it was released and accepting Bitcoin donations for DDNet. But I have always used those donations pretty much immediately, and thus I have no investment in cryptocurrencies either way and am thus impartial to their fate.

The Good

I consider the Bitcoin idea to be quite interesting as an electronic cash system that doesn’t require any trusted intermediary. This is a small and simple idea that combines old cryptographic ideas into something new.

The job I got offered had a significantly higher salary attached than my current job. The new job would have been remote, thus enabling me to spend more time with my girlfriend and live in any location I wish.

I would have been able to work in a nice programming language, developing a blockchain implementation from the ground up with a small group of colleagues.

The Bad

Proof of work in the form of mining is currently the most common form of trustless consensus in blockchain systems. It requires immense amounts of computational hardware in the form of ASICs and GPUs as well as energy to supply them. The only function of this is to prevent any one actor to take over more than 50% of the mining market and thus controlling the currency. This is a large price to pay for replacing trust.

Looking at it the other way: Trust can be seen as a shortcut that humanity has used for a long time in order not to require expensive proof of work. For most applications this still works fine.

Companies are doing initial coin offerings (ICOs), giving out their own cryptotoken to public investors, circumventing a proper IPO process. While companies often claim that their tokens will have another purpose, this is usually not true.

Reading about blockchain is horrible because of all the hype, cargo cult and politics surrounding it. Everyone who is talking positively about cryptocurrencies and making hyperbolic claims seems to have invested in them. So they might be trying to pump up the value of cryptocurrencies in order to profit themselves. It might be that this aspect is so dominant that the actual technological details are less relevant. It feels like talking to a religious group that is trying to convert you to the one true faith that will solve all problems you and society have. I wish it was possible to filter out everyone who had money invested in the fate of cryptocurrencies in those discussions.

It seems like people already know that they want to use a blockchain even before they understand what the problem is, basically a solution in search of problems. The main use of cryptocurrencies right now seems to be as a speculation device.

The Conclusion

In the end I guess I don’t want to be associated with the current state of the blockchain ecosystem. So even if the job offer is great, I have to deny it. It took me a long time to realize this for myself and getting my priorities straight.

Discussion on Hacker News.

HANA C++ Development Environment and Processes

2017-12-08T00:00:00-05:00

I started working as a C++ developer in the HANA Core Platform team at SAP in Walldorf, Germany more than a year ago. In this time I have gotten some insights into the development environment and processes. I will use this post to illustrate them by the example of adding a small new feature and explaining the steps on the way of getting it released. Some of this will be specific to HANA, some to my team inside HANA, some to my own system.

Compilation

HANA is the in-house database powering many of SAP’s products. It is written in C++ with Python tests, and the entire code base lives inside of a single git repository. Hundreds of developers from all around the world are developing on about 10 million lines of C++ code and 15 million lines of Python tests.

Since HANA is deployed on Linux exclusively, many developers are using Linux on their workstations as well. So far Windows is still supported as a development environment, but this will change in 2019, leaving Linux as the only choice. During day-to-day work you still get to interact with Windows a bit (more than I would prefer), since the Laptop has Windows with the traditional Microsoft Office products on it. But since the actual development happens on Linux, I am happy enough, being able to use the xmonad window manager and software environment I have gotten used to over the last decade.

HANA is deployed on some rather impressive machines, so in order to test the code adequately the developer workstations are quite beefy as well. I am typing this text from a 20 core / 40 thread Xeon E5-2660 v3 CPU with 128 GB of RAM.

Still compiling HANA is no quick feat. Expect something around 2 hours to build it from scratch on your local machine (illustrated below), and be prepared for some heat output, which is especially noticeable in summer. About half of the colleagues have chosen to move their workstations to the data center and access it remotely instead of working on it locally. Since they are still in one of our SAP buildings in Walldorf, the latency is low enough that a direct X connection is fast enough to be nearly undistinguishable from a local application.

So if even your beefy developer machine is not enough, how do you compile the product faster? Obviously by combining all of the developer machines together in a compile cluster:

Using the compile cluster the build time gets reduced to 16 minutes and now linking takes half of that time. Right now we are still using the gold linker, but after bug fixes lld should cut down the linking time significantly.

Coding

But for the code in my team we have another approach. Instead of running the full database and testing using Python tests we have extensive unit tests, literally covering our entire code. Every single line is covered and about 98% of all regions. In order to submit any new code you have to write a unit test that will cover this exact code and include it in the same change.

Expect 5 minutes to build and run the unit tests from scratch, while building and running them after a change can be as fast as 15 seconds if you didn’t change much and thus not much has to be recompiled.

People use quite a variety of IDEs and code editors, with Sublime Text, Qt Creator, Eclipse, Visual Studio, emacs and vim being popular choices (in no particular order). Personally I like to work directly on terminals, so I use vim with the YouCompleteMe semantic code completion (and go to definition, type analysis and some more) based on clang/llvm. I use a custom YCM version which also shows me the size and alignment of a variable along with its type:

plangen::PipelineBuilder & => hex::plangen::PipelineBuilder & (size: 128 B, align: 16 B)

Using vim and other command line tools gives me the advantage that I can work just fine as long as I have an SSH connection to my workstation, nothing else required.

Let’s start by coming up with a useful new feature, tracing the name of the DummyNoRun operator:

#pragma once

#include "hex/planex/framework/OperatorBases.hpp"
#include "hex/planex/planviz/HasReversedConnection.hpp"

#include "warnings/clang_warnings_hard.h"
#include "warnings/gcc_warnings_veryhard.h"

namespace hex {
namespace operators {

class DummyNoRunOp final
    : public planviz::HasReversedConnection,
      public planex::NoRunOperator
{
    void traceOperatorName(ltt::ostream& os) const;
};

} // namespace operators
} // namespace hex

#include "warnings/clang_warnings_end.h"
#include "warnings/gcc_warnings_end.h"

The implementation is quite simple as well:

void DummyNoRunOp::traceOperatorName(ltt::ostream& os) const
{
    os << "DummyNoRunOp";
}

For HANA development we use C++14 at this moment. The only unusual thing is that we don’t use the STL, but instead have our own implementation of it, the LTT, which enforces allocator use.

Our build system wraps around cmake, so it is quite reasonable to use. After adding the cpp file to CMakeLists.txt we can locally run our unit tests and verify that nothing broke so far.

But when we’re compiling the code with hm b -b Optimized unitHexOperators (read as: HappyMake build Optimized build [with optimizations and assertions] of the unitHexOperators target) we get an unexpected result:

In file included from ../../hex/plangen/PredInitOpGenerator.cpp:14:0:
../../hex/operators/table/DummyNoRunOp.hpp:16:10: error: ‘virtual void hex::operators::DummyNoRunOp::traceOperatorName(ltt::ostream&) const’ can be marked override [-Werror=suggest-override]
     void traceOperatorName(ltt::ostream& os) const;
          ^~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
ERROR: subcommand failed

Our warning levels are quite high and all warnings are converted to errors. We even enable warnings in headers (and disable them again at the end in order to not affect other headers that might be included later). In this case the fix is quite simple:

    void traceOperatorName(ltt::ostream& os) const override;

Now our code builds and the unit tests succeed:

$ hm b -b Optimized runtests_hex
...
[==========] 728 tests from 147 test cases ran. (11617 ms total)
[  PASSED  ] 728 tests.

  YOU HAVE 2 DISABLED TESTS



Build started:	2017-11-29 16:26:40.459482
Build finished:	2017-11-29 16:26:52.988541
Elapsed time:	12.529s
Command count:	11 (0.9 per sec)
Log directory:	/home/dXXXXXX/src2/build/Optimized/hm_log/build/24
SUCCESS

We can also run the code coverage based on our unit tests locally using CheckBot. Unsurprisingly we will find out that our code is not covered yet:

###############################################################
                  CheckBot - Review Details
###############################################################

###############################################################
                      CheckBot - Summary
###############################################################

______________
General Issues
______________
[CheckHexCodeCoverageClang] (error) hex/operators/table/DummyNoRunOp.cpp: 3 of 3 lines uncovered, 0.00% coverage!
[CheckHexCodeCoverageClang] (error) Line coverage only at 99.99%
[CheckHexCodeCoverageClang] (info) Coverage report can be found at 
file://///home/dXXXXXX/src2/build/linuxx86_64-clangcov-release_hex_with_code_coverage/hexCoverageClang/index.html

CheckBot detected 3 issues (2 errors, 1 info).

	error ... CheckHexCodeCoverageClang  (681748.81 msec)

Overall score: -2

(Legend: ok = +1, info = +1, warning = -1, error = -2)

At the bottom of the report we see the “Overall score: -2”, which indicates that our change would not be allowed to be merged by Gerrit. So let’s take a closer look, inside of the linked report we find our code:

Google Test/Mock based unit tests are used to test every part of the code in our team. 100% unit test line coverage is achieved with a development version of llvm-cov. Compared to the gcc based gcov/lcov coverage tools, the LLVM based llvm-cov is in active development, runs much faster and can show you what region of a line is actually covered, allowing finer granularity.

So let’s write a unit test to satisfy the 100% line coverage requirement:

#include "hex/operators/table/DummyNoRunOp.hpp"
#include "hex/test/HexTestBase.hpp"
#include "hex/planex/test/sandbox/SimpleOperatorSandbox.hpp"

#include "warnings/clang_warnings_hard.h"
#include "warnings/gcc_warnings_hard.h"
#include "warnings/msvc_warnings_hard.h"

using namespace ::testing;

namespace hex {
namespace operators {
namespace test {

class UnitDummyNoRunOp : public HexTestBase<>
{
protected:
    planex::SimpleOperatorSandbox<DummyNoRunOp> m_sb{alloc()};

    ltt::ostringstream m_oss{alloc()};
};

TEST_F(UnitDummyNoRunOp, traceOperatorName)
{
    m_oss << planex::Operator::TraceName{m_sb.op()};
    EXPECT_THAT(m_oss.c_str(), StrEq("UnitDummyNoRunOp"));
}

} // namespace test
} // namespace operators
} // namespace hex

For more coarse-grained testing there are also JSON based component tests as well Python based integration tests available.

Local Verification

We should also make use of some static code checkers, like clang-format and clang-tidy, before we push the change for remote testing. After all, this ensures a consistent style, fixes some common problems and reduces the mental load on the manual reviewers, since they can rely on the tools instead of complaining about the same trivial nitpick on every change:

###############################################################
                  CheckBot - Review Details
###############################################################

_____________________________________________
hex/operators/table/test/UnitDummyNoRunOp.cpp
_____________________________________________

[CheckClangFormat] (info) This file is subject to a clang-format code style. CheckBot can reformat the file, see [...]

line 2,0 - 2,35:
#include "hex/test/HexTestBase.hpp"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[CheckClangFormat] (warning) 

 #include "hex/test/HexTestBase.hpp"

should be deleted

line 3,0 - 3,0:
#include "hex/planex/test/sandbox/SimpleOperatorSandbox.hpp"
^
[CheckClangFormat] (warning) 

 #include "hex/test/HexTestBase.hpp"

should be added

Luckily clang-format can even fix the problem on its own by just adding a parameter.

Remote Verification

Finally we can push the change to Gerrit. Now our local work is done and we can start working on something else. Meanwhile automated builds and tests will happen on Linux and Windows machines, as well as a coverage run and various static code analyzers (that we were too lazy to run locally).

By default just our unit tests and component tests are compiled and executed. But if you want to build the full HANA database and run the Python tests a separate profile is available for that and can be turned on in the Gerrit web interface for specific changes.

Code Review

Once we have verified that everything works fine on Linux with GCC and Clang as well as Windows with MSVC, we can add a reviewer to our change and set the change to “Ready for Review”. A second reviewer will be chosen automatically. If you are relatively sure that your change will survive the tests and don’t feel like waiting for them, you can of course also add the reviewers straight after submitting the change. But your reviewers might not be happy if they start reviewing already and meanwhile you are pushing new patch sets to fix compilation, tests as well as automatic warnings and suggestions.

For non-trivial changes further patch sets would follow to integrate the suggestions by the reviewers, until everyone is (reasonably) happy. Finally the change can be submitted and is out of our mind.

Merging

Except that there is still one thing missing. Our change is now inside of our component’s hex branch, but not yet in the global orange branch. The merge from hex to orange will run a huge number of correctness and performance tests before allowing our changes in. That’s why we only run it after we have collected a few changes. We also use a staging branch hex2orange.

Initially we merge the hex branch into hex2orange and try to merge hex2orange into orange.
Meanwhile everyone can keep developing new features as well as bug fixes on the hex branch.
If the merge from hex2orange to orange fails, a fix for the issue will be submitted to hex and can finally be cherry-picked to hex2orange.

But we don’t merge hex into hex2orange again, until we landed in the orange branch. Otherwise the new features from hex could cause new test failures while we were fixing the old ones. In the worst case we would never reach orange.

Once the colleagues give a green light on all tests, the merge goes in and we can sit and wait for bugs for our new feature to roll in.

Conclusion

I hope you found something interesting in this post. On a related note, SAP HANA is hiring right now in a few locations.

Discuss on Hacker News and r/programming.

C++ Quiz #1

2017-12-01T00:00:00-05:00

What is the output of this small snippet that is based on real C++ code?

#include <iostream>

struct Foo
{
    Foo() { std::cout << "standard" << std::endl; }
    Foo(const Foo&) { std::cout << "copy" << std::endl; }
    Foo(int) { std::cout << "int" << std::endl; }
    Foo(int, int) { std::cout << "int, int" << std::endl; }
    Foo(const Foo&, int) { std::cout << "Foo, int" << std::endl; }
    Foo(int, const Foo&) { std::cout << "int, Foo" << std::endl; }
};

void f(Foo) {}

struct Bar
{
    int m_i;
    int m_j;

    Bar()
    {
        f(Foo(m_i, m_j));
        f(Foo(m_i));
        Foo(m_i, m_j);
        Foo(m_i);
        Foo(m_i, m_j);
    }
};

int main()
{
    Bar();
}

The code compiles without warnings with clang++ 5.0.0 as well as g++ 7.2.0 using -Wall -Wextra.

Solution

The preliminaries are simple:

We have a struct Foo with a few different constructors, each of which prints a text that represents its parameter types.
We have a struct Bar with two members, m_i and m_j.
The main function calls the standard constructor of Bar.

The constructor of Bar is where the magic happens, so let’s go through it line by line:

f(Foo(m_i, m_j)) creates a temporary Foo object by calling the Foo constructor with m_i and m_j as parameters, so int, int is printed. The resulting Foo object is then passed to the function f.
f(Foo(m_i)) works analogously, it calls the Foo constructor with m_i as parameter, so int is printed. The resulting Foo object is then passed to the function f.
Foo(m_i, m_j) works the same as the first line, except it doesn’t pass the resulting temporary Foo object to a function, so it is destroyed again immediately, int, int is printed again.
So far so good, but now look closely. Foo(m_i) surprisingly behaves entirely differently from all the previous lines. It does not call the Foo constructor with m_i as the parameter. Instead it creates a Foo object of the name m_i, just like Foo m_i would. So standard is printed.
Now the last line looks just like the third line, but it still does something different. Why? Because the name m_i is no longer referring to the int m_i member of Bar. Instead m_i is now referring to a local variable of type Foo, so Foo(m_i, m_j) prints Foo, int.

Explanation

Thus spoke the C++ standard:

§8.3 Meaning of declarators

[…]

6 In a declaration T D where D has the form

( D1 )

the type of the contained declarator-id is the same as that of the contained declarator-id in the declaration

T D1

Parentheses do not alter the type of the embedded declarator-id, but they can alter the binding of complex declarators.

This kind of ambiguous parsing can lead to dangerous situations, as is documented in DCL53-CPP:

#include <mutex>

static std::mutex m;
static int shared_resource;

void increment_by_42() {
  std::unique_lock<std::mutex>(m);
  shared_resource += 42;
}

The code looks like it locks the mutex m while modifiying shared_resource, but instead a new mutex m is created, shadowing the global mutex m.

Future Directions

The upcoming Clang version will have a warning for this:

y.cpp:25:12: warning: parentheses were disambiguated as redundant parentheses around declaration of variable named
      'm_i' [-Wvexing-parse]
        Foo(m_i);
           ^~~~~
y.cpp:25:9: note: add enclosing parentheses to perform a function-style cast
        Foo(m_i);
        ^
        (       )
y.cpp:25:12: note: remove parentheses to silence this warning
        Foo(m_i);
           ^   ~

time.gif

2017-08-11T00:00:00-04:00

This is an endless GIF that always shows the current time in UTC:

Source Code

(From reports it doesn’t seem to work on Safari, other browsers should be fine.)

time.gif is written in Haskell and works by dynamically generating each frame of the GIF and slowly feeding them over the HTTP connection.

There is no guarantee that this GIF shows a reasonable time and this is just for fun anyway, so better don’t build your next project based on the time from this GIF. If my server is overloaded, you can try compiling it yourself and run it locally.

Update: Optimized, runs at less than 1% CPU with 500 simultaneous connections open, LZW encoding reduces bandwidth from 4 KB/s to 300 B/s.

-- Wait for incoming connections and start delivering a GIF to them
loop :: Int -> FrameSignal -> Socket -> IO ()
loop delay frameSignal sock = do
  (conn, _) <- accept sock

  forkIO $ body conn
  loop delay frameSignal sock

  where
    body c = do
      f <- receiveMSignal frameSignal
      sendAll c $ msg $ initialFrame (delay `div` 10000) f
      nextFrame c

    nextFrame c = do
      f <- receiveMSignal frameSignal
      sendAll c $ frame (delay `div` 10000) f
      nextFrame c

    msg content = B.intercalate "\r\n"
      [ "HTTP/1.0 200 OK"
      , "Server: gifstream/0.1"
      , "Content-Type: image/gif"
      , "Content-Transfer-Encoding: binary"
      , "Cache-Control: no-cache"
      , "Cache-Control: no-store"
      , "Cache-Control: no-transform"
      , ""
      , content
      ]

This is cannibalized from gifstream, which lets you play snake and have people watch a GIF livestream. It was actually created as a Christmas exercise for students of the Programming Paradigms course at KIT.

Discuss on Hacker News (2022-10-27) (previous thread from 2017-08-12)

Nim Code Coverage

2016-11-01T00:00:00-04:00

Creating code coverage reports with Nim is surprisingly easy. You can simply use the good old gcov and lcov tools. Nim can be told to insert its own line information with the --debugger:native command line parameter.

Here’s the small example program we’re looking at:

var x = 0
if x > 1:
  echo "foo"
echo "bar"

Note that if we change the condition to if false: the Nim compiler optimizes the impossible code away and it will not count as uncovered. The same thing can happen with entire uncovered functions when optimizations and dead code elimination are enabled.

The file is saved as x.nim. Here’s the script I use to create the code coverage report:

#!/bin/sh
rm -rf *.info html nimcache
nim --debugger:native --passC:--coverage --passL:--coverage c x
lcov --base-directory . --directory . --zerocounters -q
./x
lcov --base-directory . --directory . -c -o x.info
lcov --remove x.info "lib/*" -o x.info # remove Nim system libs from coverage
genhtml -o html x.info

You can look at the final html report generated.

A Taste of Haskell

2016-10-21T00:00:00-04:00

In this post I want to highlight a few fun aspects of the Haskell programming language. The purpose is to give you a taste of Haskell so that you will want to learn more of it. Don’t consider this as a tutorial or guide but rather as a starting point, as it is based on a short talk I held at work, which in turn is based on my favorite material from holding practical courses about Haskell at university.

Let’s start by seeing how programmers compare Haskell to a mainstream programming language, for example Java:

Image Source: Hammer Principle

Doesn’t sound that good for Java, what do people say about Haskell?

Image Source: Hammer Principle

Sounds much better and makes it seem like a reasonable endeavour to learn Haskell! So what is Haskell actually?

Haskell is a purely functional, lazily evaluated, statically typed programming language with type inference.

While this definition sounds complicated you should understand more of it after reading this post. Let’s start with the basics: What’s the difference between an imperative programming language and a functional one? In imperative programming the basic operation is changing a stored value. In functional programming the basic operation is applying a function to arguments.

Haskell itself was designed in 1990 by a scientific committee with the purpose of being a basis for functional programming research. The Glasgow Haskell Compiler (GHC) is the most popular implementation and the one we will be using. You can get it as part of the Haskell Platform as well. Haskell code can be compiled (ghc), but also interpreted (ghci, interactive).

In the rest of this blog post we will look at some key features of Haskell interactively, so you can get your own installation of GHC and follow along and experiment with the code.

Arithmetic

I belive that the best way to learn a programming language is by playing around with it. So that’s what we’re going to do now. I’ll show a few examples and explain the cool Haskell features that we encounter on the way.

We start out by running ghci, the interactive Glasgow Haskell Compiler interpreter (indicated by λ>). We also create a single file tut.hs which we will use to write some more advanced Haskell code and load in the interpreter.

For starters let’s do some basic arithmetic in GHCi, replacing our dusty old calculator:

λ> 2+3
5
λ> 4 * 5
20
λ> 5 / 6
0.8333333333333334
λ> 2 ^ 10
1024
λ> 2 ^ 1000
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376

We see that Integers can be of unbounded size, instead of the usual limits of 32 or 64 bits in many programming languages. Of course if you really want them there are also more efficient machine ints in Haskell:

λ> (2 :: Int) ^ 10
1024
λ> (2 :: Int) ^ 100
0

Using the :: Int notation we specifiy that the number 2 is explicitly of type Int instead of being automatically inferred to be of type Integer. While we’re looking at types, there are also floating point numbers of course:

λ> 2 ** 10
1024.0
λ> 2 ** 1000
1.0715086071862673e301
λ> 2 ** 10000
Infinity

As usual, floating point numbers are of limited precision, so at some point we just reach “approximately infinity”. Let’s do some more math:

λ> pi
3.141592653589793
λ> sin

<interactive>:12:1:
    No instance for (Show (a0 -> a0))
      (maybe you haven't applied enough arguments to a function?)
      arising from a use of ‘print’
    In the first argument of ‘print’, namely ‘it’
    In a stmt of an interactive GHCi command: print it

Looks like sin is a function, but the error message is already confusing. Looking at the part in brackets gives us a good hint: maybe you haven’t applied enough arguments to a function?. Let’s check out the type of sin:

λ> :type sin
sin :: Floating a => a -> a

Invoking :type tells us that sin is a function that takes a value of type a and returns a value of type a, where a is a floating point number. So let’s pass a number to sin:

λ> sin 0.5
0.479425538604203
λ> sin 10
-0.5440211108893698
λ> div 15 6
2
λ> :t div
div :: Integral a => a -> a -> a

div is another function. As you can see it accepts two parameters, both of type a and finally returns a value of type a. In this case a has to be an Integral, some kind of integer-like type. We can even ask GHCi what an Integral is supposed to be:

λ> :info Integral
class (Real a, Enum a) => Integral a where
  quot :: a -> a -> a
  rem :: a -> a -> a
  div :: a -> a -> a
  mod :: a -> a -> a
  quotRem :: a -> a -> (a, a)
  divMod :: a -> a -> (a, a)
  toInteger :: a -> Integer
  	-- Defined in ‘GHC.Real’
instance Integral Word -- Defined in ‘GHC.Real’
instance Integral Integer -- Defined in ‘GHC.Real’
instance Integral Int -- Defined in ‘GHC.Real’

Integral is a type class which requires a few functions to be defined for the type, like quot and rem. We also see that an Integral type needs to have all properties of a Real and an Enum type. Three types are known to GHCi which adhere to this type class: Word, Integer and Int

It looks a bit awkward to write div 15 6, so Haskell offers some syntactic sugar to use the div function in infix notation:

λ> 15 `div` 6
2
λ> 15 `mod` 6
3

The same thing works in reverse to use operators as regular functions, using brackets:

λ> (+) 2 3
5
λ> :t (+)
(+) :: Num a => a -> a -> a

Of course there are quite a lot of other functions that we won’t have time to explore, but you can always explore them yourself in the documentation or using Hoogle:

λ> max 2 3
3
λ> max 5 3
5
λ> 2 > 3
False
λ> not (2 > 3)
True
λ> True && False
False
λ> True && True
True

Lists

Lists are the most important data structure for us, they are simply a collection of values of the same type:

λ> :info []
data [] a = [] | a : [a] 	-- Defined in ‘GHC.Types’
...

What this definition tells us is that a list is a data type that is either an empty list ([]) or a value concatenated with a list itself (a : [a]). So the data type is recursively defined, referring to itself. With this knowledge we can create basic lists ourselves:

λ> []
[]
λ> 1:[]
[1]
λ> 1:2:[]
[1,2]

We can also use some syntactic sugar to create lists instead:

λ> [1,2,3,4,5]
[1,2,3,4,5]

We just said that a list is supposed to be a collection of values of the same type. What happens if we try to break that?

λ> [1,2,3,4,5,True]

<interactive>:27:2:
    No instance for (Num Bool) arising from the literal ‘1’
    In the expression: 1
    In the expression: [1, 2, 3, 4, ....]
    In an equation for ‘it’: it = [1, 2, 3, ....]

Good, that shouldn’t work and indeed it doesn’t. The error message tells us that the list is assumed to be a list of booleans because of the final value. But 1 is not a boolean value, so there is no valid type for this list.

λ> [1..5]
[1,2,3,4,5]
λ> [1,3..20]
[1,3,5,7,9,11,13,15,17,19]

We can store values in variables, but the name “variable” might confuse you a bit, because variables can not be overwritten:

λ> let xs = [1..5]
λ> let xs = [1..6]

The seconds line creates a new variable xs that makes the old one invisible in the new scope. Actually variables are always immutable in Haskell. That means you can easily share access to the same data because there is no way in which it can be overwritten:

λ> let ys = xs

Let’s look at a few common list operations in Haskell:

λ> head xs
1
λ> tail xs
[2,3,4,5,6]
λ> init xs
[1,2,3,4,5]
λ> last xs
6
λ> take 2 xs
[1,2]
λ> length xs
6

Image Source: Learn You a Haskell for Great Good!

Knowing how lists are implemented we can easily implement our own definitions of these functions:

-- data [] a = [] | a : [a]

head' (x:xs) = x

This definition uses pattern matching. The pattern is (x:xs) where x and xs are variables delimited by the cons (:). We use the knowledge of the definition to split up the passed value into two parts, the first element x and the rest of the list xs. Then the result of our function is simply the first element x.

λ> :l tut
λ> head' [1..5]
1
λ> head' []
*** Exception: tut.hs:1:1-16: Non-exhaustive patterns in function head'

Oops, we didn’t cover one possible pattern, the empty list! We can simply add a definition for that to throw a nicer error message:

head' [] = error "head of empty list undefined"

λ> :r
[1 of 1] Compiling Main             ( tut.hs, interpreted )
Ok, modules loaded: Main.
λ> head' []
*** Exception: head of empty list undefined

Unfortunately there is nothing usefuly we can return for head [], because it’s impossible for us to construct a value of any arbitrary type. Let’s look at the type of head':

λ> :t head'
head' :: [t] -> t

The type was automatically inferred for us, but we can also write it down explicitly if we want to make sure that we don’t break it in the future:

head' :: [a] -> a

This means that head' is a function that takes a list of values of type a as its parameter and returns a single value of type a.

If we had defined head’ for integer lists only, we could return a 0 for example as the default value:

head' [] = 0
head' (x:xs) = x

But if we now check the inferred type we notice that it changed, now only lists containing numbers are supported:

λ> :t head'
head' :: Num a => [a] -> a

So in this case there is nothing better we can do than throw an error.

Analogously we can define the tail function:

tail' [] = []
tail' (x:xs) = xs

λ> tail' [1..5]
[2,3,4,5]
λ> tail' []
[]

This was easy enough. How can we define the last function?

λ> last [1..5]
5

last' [] = error "last of empty list undefined"
last' [x] = x
last' (x:xs) = last' xs

Here the idea is to reduce the problem to something that we can solve. If the list only contains a single element, we know that this exact element is the last one. If the list has more than one element, we remove the first element of the list and recursively call last' on the rest of the list. Yet again the last element of an empty list makes no sense, so we throw an error.

What about defining init? The list of all values but the last?

λ> init [1..5]
[1,2,3,4]

init' [] = error "init of empty list undefined"
init' [x] = []
init' (x:xs) = x : init' xs

The idea here is a bit more complicated. We know that init of a list with one element [x] is the empty list []. If the list has more than one element we know that init contains the first element x, concatenated with init of the rest of the list.

Finally let’s define the length function, which works similarly:

length' [] = 0
length' (x:xs) = 1 + length' xs

The length of an empty list is 0. The length of a list with more than 0 elements is 1 plus the length of the rest of the list. These functions read like mathematical definitions of the properties they are encoding.

Higher-order Functions

So we have defined our first basic functions and noticed that there is no magic happening in Haskell’s standard library. Instead all of these functions are easily implementable. Now let’s look at some more advanced operations that we can perform on lists, for example mapping a function to a list, thus applying it to each value in the list:

λ> let f x = 2 * x
λ> map f [1..5]
[2,4,6,8,10]

It’s a common theme in functional programming to pass functions as parameters to higher order functions. But it’s a bit annoying to define a named function explicitly all the time, so instead we can quickly create an unnamed function, called a lambda function, instead:

λ> map (\x -> x * 2) [1..5]
[2,4,6,8,10]

You can see the lambda function (\x -> x * 2), which is specified to take a parameter x and return x * 2. Of course Haskell has some sweet syntactic sugar to do this even more succinctly by just writing (*2):

λ> map (*2) [1..5]
[2,4,6,8,10]

Let’s see how we can define our own map function:

map' f [] = []

The base case is easy: When we get an empty list passed, the result is also an empty list.

map' f (x:xs) = f x : map' f xs

Otherwise we take the first element x in the list, apply the function f to it and create a new list with this new value as the initial element. We recurse and apply map to the rest of the list in the same way.

At this point we can take a look at the actual implementations in GHC’s standard library and we notice that map is implemented exactly as we wrote it.

Note that we’re not modifying the passed data structure directly, instead we create a new one. Actually in Haskell there is no way to modify data structures, they are all immutable. And functions are pure, so there is no way for a function to have any side effects other than returning a value directly. That’s also why we have referential transparency: When you call a function with the same inputs, it will always return the same output.

Let’s turn to the next function, filter, which keeps only those elements in a list which fulfil a predicate:

λ> filter odd [1..5]
[1,3,5]

Filter is also easy to implement:

filter' f [] = []
filter' f (x:xs)
  | f x       = x : filter' f xs
  | otherwise =     filter' f xs

The guarded equation at the start of the line means that we check a boolean. When f x is true we return the first line, including x, otherwise we don’t include x in the rest of the list. Finally we recurse with the rest of the list, until we reach the base case for the empty list.

By implementing a few functions on list a common pattern emerges. Let’s see how to implement the sum over a list:

λ> sum [1..5]
15

sum' [] = 0
sum' (x:xs) = x + sum' xs

We already implemented the length of a list:

length' [] = 0
length' (x:xs) = 1 + length' xs

How do we define the and function over an entire list?

λ> and [True, False, True]
False
λ> and [True, True, True]
True

and' [] = True
and' (x:xs) = x && and' xs

Notice the similarity by now?

The functions sum, length and and all are built in pretty much the same way. So we can create an abstraction over this pattern, which is called a right-associative fold, or foldr in short:

foldr' f i [] = i
foldr' f i (x:xs) = x `f` foldr' f i xs

Using this pattern it becomes trivial to define these functions:

sum'' xs = foldr' (+) 0 xs

Even better, we don’t need to write down the last parameter, xs:

sum'' = foldr' (+) 0

What’s going on there? Actually in Haskell when you pass a parameter to a function, a new function is returned. So you can consider each function as accepting a single parameter, then returning a new function, which is applied to the next parameter, and so on.

λ> foldr' (+) 0 [1..5]
15
λ> :t foldr'
foldr' :: (t -> t1 -> t1) -> t1 -> [t] -> t1
λ> :t foldr' (+)
foldr' (+) :: Num t1 => t1 -> [t1] -> t1
λ> :t foldr' (+) 0
foldr' (+) 0 :: Num t1 => [t1] -> t1
λ> :t foldr' (+) 0 [1..5]
foldr' (+) 0 [1..5] :: (Enum t1, Num t1) => t1

So in our definition of sum'' we don’t need to have a parameter and put it into foldr', instead we can also have no parameter and just return the function that is returned from (foldr' (+) 0), which takes a list as its parameter.

and'' = foldr' (&&) True
length'' = foldr' (\x y -> 1 + y) 0

We can even define map and filter with foldr:

map'' f = foldr' (\x xs -> f x : xs) []

filter'' f = foldr' go []
  where go x xs
          | f x = x : xs
          | otherwise = xs

While working with lists in Haskell you might wonder what happens if we never reach the base case? We have lazy evaluation in Haskell, which tells us that a data structure is only evaluated when it’s actually needed. So we have no problem handling lists of infinite size:

λ> let x = [1..]
λ> take 5 x
[1,2,3,4,5]
λ> take 5 $ map (*2) x
[2,4,6,8,10]
λ> take 5 $ map (*2) $ filter odd x
[2,6,10,14,18]
λ> x
...

We can even combine multiple infinite lists without any problem:

λ> let y = [1,3..]
λ> let z = zipWith (+) x y
λ> take 10 z
[2,5,8,11,14,17,20,23,26,29]

Of course length won’t work as it will never reach the base case:

λ> length z
^CInterrupted.

Let’s go a step back and create an infinite list of ones:

ones = [1,1..]

The problem is that this is very inefficient, because we recalculate new elements all the time and have to store them in memory.

λ> length ones

You can easily see how the memory usage grows. A more efficient solution is to use recursion in the definition:

ones' = 1 : ones'

We can use ones' itself in the definition of ones', just referring to it. Thanks to lazy evaluation the next value is only evaluated when it is needed, so when we call:

λ> take 3 ones'
[1,1,1]

We see that 1 is the first value of the list, we recurse into ones', we see that 1 is the next value we get, we recurse into ones' and finally we get the last 1.

λ> length ones'

Of course there is no useful result, it’s an infinite calculation, but at least we don’t create a list of infinite size, instead referring to ourselves.

Let’s look at another fun list combinator that takes two lists and creates a new one out of them:

λ> :t zipWith
zipWith :: (a -> b -> c) -> [a] -> [b] -> [c]
λ> zipWith (+) [1..10] [100..110]
[101,103,105,107,109,111,113,115,117,119]

So, zipWith is a function that takes a function (a -> b -> c) as its first parameter. The second parameter is a list of values of type a, the third parameter is a list of values of type b and finally a list of type c is returned.

Of course to understand it we best implement it:

zipWith' f (x:xs) (y:ys) = f x y : zipWith' f xs ys
zipWith' f xs ys = []

We can use zipWith to trivially define a list of all fibonacci numbers:

fibs = 0 : 1 : zipWith' (+) fibs (tail fibs)

λ> take 10 fibs
[0,1,1,2,3,5,8,13,21,34]

The start makes sense, a list of fibs starts with 0 and 1, then the rest is defined with zipWith over fibs itself and (tail fibs). This smells like magic, how can it possibly work? Functions are pure in Haskell. That means that a function always returns the same value when you call it with the same arguments. There is no way to have a variable in which you store some state. Since data structures and functions in Haskell are immutable and pure we can not change them in any way, so we can reuse them without having to recalculate them. So in this case we can refer to fibs' multiple times even inside the definition of fibs' itself.

fibs        0 : 1 : 1 : 2 : 3 : 5 : 8
tail fibs   1 : 1 : 2 : 3 : 5 : 8 :
zipWith (+) 1 : 2 : 3 : 5 : 8 :   :

List Comprehensions

Instead of all these filters and maps we can also use list comprehensions:

λ> map (*2) $ filter odd [1..5]
[2,6,10]
λ> [x*2 | x <- [1..5], odd x]
[2,6,10]

We can even implement the sieve of eratosthenes to calculate all prime numbers as a oneliner with a list comprehension:

sieve (p:xs) = p : sieve [x | x <- xs, x `mod` p > 0]
primes = sieve [2..]

Because in Haskell functions are pure and data is immutable we can perform equational reasoning. We can guarantee that code can be replaced by its definition. Basically this means that you can do refactoring without any risks. You know that nothing can go wrong because there is no state. Every function only takes its arguemnts and returns a value based on those, always the same value. So you can reorganize your functions however you want, the order does not matter and there is actually no order semantically enforced in Haskell.

  primes
= sieve [2..]
= 2 : sieve [x | x <- [3,4,5,6,7,8,9,10,11..], x `mod` 2 > 0]
= 2 : sieve [3,5,7,9,11..]
= 2 : 3 : sieve [x | x <- [5,7,9,11..], x `mod` 3 > 0]
= 2 : 3 : sieve [5,7,11..]
= 2 : 3 : 5 : sieve [...]

Conclusion

I hope you enjoyed this small excursion into functional programming land with Haskell. Maybe you learned something that gives you something to think when programming in your favorite programming language. If you’re interested in learning more Haskell, here are a few books you can read:

Discussions on Hacker News and r/Programming.

Writing a 2D Platform Game in Nim with SDL2

2016-06-14T00:00:00-04:00

Japanese Translation

In this article we’re going to write a simple 2D platform game. You can also consider this as a tutorial for game development with SDL2 in Nim.

We will read in user input, display graphics and a tile map, and simulate simple 2D physics with collision detection and handling. Afterwards we will implement simple camera movement and game logic. To display some information we will render texts and develop a caching mechanism for said text rendering.

The final result will be a binary file that requires only SDL2 and can be easily distributed, perfect for games. If you’re on Linux we will also present a simple way to cross-compile Nim programs for Windows.

For the sake of simplicity we’re going to use the familiar graphics from DDNet and Teeworlds, with the end result of this tutorial looking like this:

We’re going to follow along with the development throughout this article with illustrative images and videos, but the best way to learn is if you follow along yourself by implementing the steps from this article. The code is purposefully kept simple and easy to extend so that you can play around with it and try out all kinds of changes to get an intuitive understanding. At the end of every section there is a link to its full source code.

The iterations of the code of this article and the final result are available in a repository on GitHub. The resulting binaries can be downloaded here: Win64, Win32, Linux x86_64, Linux x86

Preliminaries

For this post we require:

The Nim programming language and its package manager Nimble
SDL 2, SDL_image 2, SDL_ttf 2 (all for developers)
Nim SDL2 wrapper and strfmt, which can be installed using Nimble

On a unixoid system like Linux or Mac OS X the installation looks something like this:

# Debian / Ubuntu
$ sudo apt-get install git libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev
# Arch Linux
$ pacman -S git sdl2 sdl2_image sdl2_ttf
# Homebrew on OS X
$ brew install git sdl2 sdl2_image sdl2_ttf
# FreeBSD
$ pkg install git sdl2 sdl2_image sdl2_ttf

$ wget http://nim-lang.org/download/nim-0.14.2.tar.xz
$ tar xvf nim-0.14.2.tar.xz
$ cd nim-0.14.2
$ make -j4
$ echo 'export PATH=$HOME/nim-0.14.2/bin:$PATH' >> ~/.profile
$ source ~/.profile

$ git clone https://github.com/nim-lang/nimble.git
$ cd nimble
$ nim -d:release c -r src/nimble install
$ echo 'export PATH=$HOME/.nimble/bin:$PATH' >> ~/.profile
$ source ~/.profile

$ nimble install sdl2 strfmt

Note that you also need a C compiler on your system, preferably GCC or Clang.

Instead of compiling Nim and Nimble from source code you could also use your package manager to install Nim and Nimble if they’re available in a recent version.

For setting up SDL2 on other platforms more extensive guides exist, as do for Nim and nimble.

1. First Running Program

If you have seen SDL2 programs written in C or C++ before, you will notice that what we’re doing in Nim is very similar. Actually Nim’s SDL2 wrapper is just a thin layer wrapping the original SDL2 interface from C. This has the advantage that what you learn from any SDL2 tutorial is applicable, but the disadvantage is that you end up with a bit more boilerplate than with a more high-level Nim library.

We start with exactly this boilerplate to initialize our window:

import sdl2

type SDLException = object of Exception

template sdlFailIf(cond: typed, reason: string) =
  if cond: raise SDLException.newException(
    reason & ", SDL error: " & $getError())

proc main =
  sdlFailIf(not sdl2.init(INIT_VIDEO or INIT_TIMER or INIT_EVENTS)):
    "SDL2 initialization failed"

  # defer blocks get called at the end of the procedure, even if an
  # exception has been thrown
  defer: sdl2.quit()

  sdlFailIf(not setHint("SDL_RENDER_SCALE_QUALITY", "2")):
    "Linear texture filtering could not be enabled"

  let window = createWindow(title = "Our own 2D platformer",
    x = SDL_WINDOWPOS_CENTERED, y = SDL_WINDOWPOS_CENTERED,
    w = 1280, h = 720, flags = SDL_WINDOW_SHOWN)
  sdlFailIf window.isNil: "Window could not be created"
  defer: window.destroy()

  let renderer = window.createRenderer(index = -1,
    flags = Renderer_Accelerated or Renderer_PresentVsync)
  sdlFailIf renderer.isNil: "Renderer could not be created"
  defer: renderer.destroy()

  # Set the default color to use for drawing
  renderer.setDrawColor(r = 110, g = 132, b = 174)

  # Game loop, draws each frame
  while true:
    # Draw over all drawings of the last frame with the default
    # color
    renderer.clear()
    # Show the result on screen
    renderer.present()

main()

We introduced an sdlFailIf template that checks a condition and if the condition is true, raises an SDLException with additional error information from SDL. In the main proc we initialize SDL2, and create a regular window and an accelerated 2D renderer. Error handling is done with the sdlFailIf proc that we introduced.

For now the game loop just clears the window and draws it every frame. If you have VSync enabled and your screen is set to 60 Hz the loop will be executed 60 times per second.

We can compile and run in the same step by executing nim -r c platformer, assuming you called the file platformer.nim. To compile with optimizations use nim -d:release -r c platformer. The result is a simple one-colored window:

We can exit our small program by pressing Ctrl-C in the terminal window. Unfortunately we can’t exit it in the game window itself yet, so let’s fix that.

Full code for section 1

2. User Input

First let’s add an Input type to store all the inputs we want to support, and store an array of Inputs in our game state object:

type
  Input {.pure.} = enum none, left, right, jump, restart, quit

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr

By choosing a ref type for the Game state type we have an easy way to prevent accidentally creating copies of it. By default only the garbage collected pointer to our Game object is passed around. The inputs field is an array mapping from Input to bool, signifying which input is currently pressed (true) or not (false). Creating a new game state object is trivial, we just create a new heap object for now and assign the SDL2 renderer that we will need later:

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

We don’t need to initialize the inputs field in any way as everything is initialized to binary null by default, which is exactly what we want: Every input is set to off in the start. If we didn’t initialize the renderer field it would be a null pointer and we would get into trouble if we accidentally dereference it.

The next thing we need is a procedure that maps keyboard scan codes to our recognized inputs:

proc toInput(key: Scancode): Input =
  case key
  of SDL_SCANCODE_A: Input.left
  of SDL_SCANCODE_D: Input.right
  of SDL_SCANCODE_SPACE: Input.jump
  of SDL_SCANCODE_R: Input.restart
  of SDL_SCANCODE_Q: Input.quit
  else: Input.none

Note that toInput returns Input.none for all undefined cases. We will use this behaviour to ignore unused keyboard inputs without the need for a branch in our code. You could easily recognize multiple scan codes to map to a single input.

We modify the game loop to react to keyboard inputs by calling our new handleInput proc. We also split out the rendering itself to keep the separation of concerns clear in the main proc:

proc handleInput(game: Game) =
  var event = defaultEvent
  while pollEvent(event):
    case event.kind
    of QuitEvent:
      game.inputs[Input.quit] = true
    of KeyDown:
      game.inputs[event.key.keysym.scancode.toInput] = true
    of KeyUp:
      game.inputs[event.key.keysym.scancode.toInput] = false
    else:
      discard

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Show the result on screen
  game.renderer.present()

var game = newGame(renderer)

# Game loop, draws each frame
while not game.inputs[Input.quit]:
  game.handleInput()
  game.render()

Now we can either press q or use the close button on the window to close the game. Other kinds of user input are just stored in the inputs array for now and we will be able to use it later.

Note that the inputs array is intentionally a simple array, so that access to it is as performant as can be. If we used a hash table or some other data structure such a guarantee would not be that easy to make. Simplicity can often be beneficial and make it easier to understand what is going on in the system you’re developing.

Full code for section 2

3. Displaying Graphics

If we want to do something of interest with those user inputs we need to start displaying something other than a blue sky.

We extend our game state object to also store the player texture as well as the current position and velocity, for which we use the basic2d module:

import basic2d

type
  Player = ref object
    texture: TexturePtr
    pos: Point2d
    vel: Vector2d

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    player: Player
    camera: Vector2d

We’re going to use Teeworlds’ default tee graphic for our player:

You can save this file as player.png to follow along. If you feel funny you can also select one of hundreds of skins from the DDNet Skin Database, for example:

Don’t forget to call your player graphic player.png if you want to use an alternative one.

First we have to load the player graphic, whichever one you decided to use, in our definition of newGame and initialize the Game and Player data structures:

import sdl2/image

proc restartPlayer(player: Player) =
  player.pos = point2d(170, 500)
  player.vel = vector2d(0, 0)

proc newPlayer(texture: TexturePtr): Player =
  new result
  result.texture = texture
  result.restartPlayer()

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer
  result.player = newPlayer(renderer.loadTexture("player.png"))

restartPlayer is used to reset the player to its start position. The loadTexture procedure loads the PNG image into memory as an SDL2 texture that we can store in the Game object.

We also shouldn’t forget to initialize the SDL2 image module in our main proc, similarly to the SDL2 initialization:

const imgFlags: cint = IMG_INIT_PNG
sdlFailIf(image.init(imgFlags) != imgFlags):
  "SDL2 Image initialization failed"
defer: image.quit()

We only need support for PNG files, otherwise we could also add JPEG files with const imgFlags: cint = IMG_INIT_PNG or IMG_INIT_JPG.

Next our task is to put this together nicely. Of course the intention of the flexible player images is that parts of the body can move independently, but for the sake of simplicity we will put them into fixed positions. A simple addition would be to make the feet move in a rotating motion depending on the horizontal position of the player. Another addition would be to make the eyes follow the mouse cursor.

proc renderTee(renderer: RendererPtr, texture: TexturePtr,
               pos: Point2d) =
  let
    x = pos.x.cint
    y = pos.y.cint

  var bodyParts: array[8, tuple[source, dest: Rect, flip: cint]] = [
    (rect(192,  64, 64, 32), rect(x-60,    y, 96, 48),
     SDL_FLIP_NONE),      # back feet shadow
    (rect( 96,   0, 96, 96), rect(x-48, y-48, 96, 96),
     SDL_FLIP_NONE),      # body shadow
    (rect(192,  64, 64, 32), rect(x-36,    y, 96, 48),
     SDL_FLIP_NONE),      # front feet shadow
    (rect(192,  32, 64, 32), rect(x-60,    y, 96, 48),
     SDL_FLIP_NONE),      # back feet
    (rect(  0,   0, 96, 96), rect(x-48, y-48, 96, 96),
     SDL_FLIP_NONE),      # body
    (rect(192,  32, 64, 32), rect(x-36,    y, 96, 48),
     SDL_FLIP_NONE),      # front feet
    (rect( 64,  96, 32, 32), rect(x-18, y-21, 36, 36),
     SDL_FLIP_NONE),      # left eye
    (rect( 64,  96, 32, 32), rect( x-6, y-21, 36, 36),
     SDL_FLIP_HORIZONTAL) # right eye
  ]

  for part in bodyParts.mitems:
    renderer.copyEx(texture, part.source, part.dest, angle = 0.0,
                    center = nil, flip = part.flip)

The exact numbers are not so important, they are just how the player is meant to be put together. With renderTee we define which body part is drawn at which position and in which order. Finally each of these body parts is drawn with the SDL2 renderer using copyEx.

Now drawing the tee in our game loop is simply a call to renderTee away:

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  # Show the result on screen
  game.renderer.present()

Finally we have some visual progress again, look at the player floating in the sky:

Full code for section 3

4. Tile Map

Now that we have a working rendering system for the player, we need a map to play in. This requires us to store a texture as well as a list of tiles:

type
  Map = ref object
    texture: TexturePtr
    width, height: int
    tiles: seq[uint8]

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    player: Player
    map: Map
    camera: Vector2d

Each tile is defined to be a uint8, which means a value between 0 and 255 inclusively. Conveniently the tileset graphic for from Teeworlds have 16 × 16 = 256 tiles. We will use the grass tileset:

Download and save this image as grass.png.

To initialize the Map data structure we shall write a little parser that parses maps of this format:

0  0  0 78  0  0  0  0  0  0  0  0  0  0
5  0  0 78  0  0  0  0  0  0  0  0  0  0
21  0  0 78  0  0  0  0  0  0  0  0  0  0
21  0  0 78  0  0  0  0  0  0  0  0  4  5
21  0  0 78  0  0  0  0  0  0  0  0 20 21
21  0  0 78  0  0  0  0  0  0  0  0 20 21
21  0  0 78  0  0  4  5  0  0  0  0 20 21
21  0  0 78  0  0 20 21  0  0  0  0 20 21
38  0  0 78  0  0 22 38  0  0  0  0 22 38
49 16 16 16 16 16 48 49 16 16 16 16 48 49
52 52 52 52 52 52 52 52 52 52 52 52 52 52

Our goal in this section is for this map with the grass tileset to result in this rendering:

Each number denotes the tile that is chosen from the grass tileset. We will use this default.map for the rest of this article. Our parser is implemented in newMap and looks like this:

import strutils

proc newMap(texture: TexturePtr, file: string): Map =
  new result
  result.texture = texture
  result.tiles = @[]

  for line in file.lines:
    var width = 0
    for word in line.split(' '):
      if word == "": continue
      let value = parseUInt(word)
      if value > uint(uint8.high):
        raise ValueError.newException(
          "Invalid value " & word & " in map " & file)
      result.tiles.add value.uint8
      inc width

    if result.width > 0 and result.width != width:
      raise ValueError.newException(
        "Incompatible line length in map " & file)
    result.width = width
    inc result.height

We split the file by line as well as by word. Each number is parsed to an unsigned integer and checked whether it is in the valid range of 0..255. The final width and height are calculated from the line length and number of lines. Errors in the map data cause an exception to be thrown, quitting our game.

We have to extend newGame to initialize the map now:

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer
  result.player = newPlayer(renderer.loadTexture("player.png"))
  result.map = newMap(renderer.loadTexture("grass.png"),
    "default.map")

At this point we have the texture and the tiles of the map. What’s missing is actually rendering all of this:

const
  tilesPerRow = 16
  tileSize: Point = (64.cint, 64.cint)

proc renderMap(renderer: RendererPtr, map: Map, camera: Vector2d) =
  var
    clip = rect(0, 0, tileSize.x, tileSize.y)
    dest = rect(0, 0, tileSize.x, tileSize.y)

  for i, tileNr in map.tiles:
    if tileNr == 0: continue

    clip.x = cint(tileNr mod tilesPerRow) * tileSize.x
    clip.y = cint(tileNr div tilesPerRow) * tileSize.y
    dest.x = cint(i mod map.width) * tileSize.x - camera.x.cint
    dest.y = cint(i div map.width) * tileSize.y - camera.y.cint

    renderer.copy(map.texture, unsafeAddr clip, unsafeAddr dest)

This is similar to renderTee but uses fixed size parts of the texture. The texture is cut into tiles of size 64 × 64 pixels with 16 tiles per line. We iterate over each tile in the map and render the tile tileNr from our map texture with. Tile 0 is the air tile and is always empty so we don’t need to render it, which improves performance as typical maps are in large parts empty.

Finally we have to render our map in the main proc after rendering the player, so that it is put on top of the player:

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)
  # Show the result on screen
  game.renderer.present()

For now we don’t have a moving camera, so we use the static game.camera to get a fixed rendering position. But for now we can’t even move, so we should finally do something with the user input and implement a simple physics model.

The end result of this section features our beautiful map rendering:

Full code for section 4

5. Physics and Collisions

For our game physics we decide to have 50 ticks per second. We will only calculate the next iteration of the game when a new tick has arrived, independent of whether our game runs at 60 fps or even 240 fps. Let’s add the ticks to our main proc:

import times

proc physics(game: Game) =
  discard

var
  startTime = epochTime()
  lastTick = 0

while not game.inputs[Input.quit]:
  game.handleInput()

  let newTick = int((epochTime() - startTime) * 50)
  for tick in lastTick+1 .. newTick:
    game.physics()
  lastTick = newTick

  renderer.render(game)

That’s our physics framework, but so far the physics does nothing. We can start by adding gravity:

proc physics(game: Game) =
  game.player.vel.y += 0.75
  game.player.pos += game.player.vel

Well, there goes our player. I guess we also need to be able to restart the player now, so that we can see this amazing animation again and again just by pressing r:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  game.player.vel.y += 0.75
  game.player.pos += game.player.vel

This looks exactly like the previous GIF on repeat, so you can imagine it or click on play a few times.

Moving left and right with a and d as well as jumping with space is now easy to implement:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  if game.inputs[Input.jump]:
    game.player.vel.y = -21

  let direction = float(game.inputs[Input.right].int -
                        game.inputs[Input.left].int)

  game.player.vel.y += 0.75
  game.player.vel.x = clamp(
    0.5 * game.player.vel.x + 4.0 * direction, -8, 8)

  game.player.pos += game.player.vel

The specific values are determined by trial and error. You can change them around if you have other preferences. Note that we don’t set the player position directly, instead we modify the velocity vector and add that to the position. This is important for collision detection.

While I tried to move around as if the walls and ground had an effect, I probably failed at deceiving you and you noticed that we still don’t have any collision detection and handling. This code is largely adapted from Teeworlds. It works by checking for horizontal and vertical collisions with a tile in moveBox, which manipulates the player’s position based on the passed velocity vector.

When a collision occurs the player is moved just out of the tile in the right direction. For the sake of simplicity in isSolid we consider every tile other than air, start, finish a solid block. Floating point player positions are converted to indices in the tile map.

import math

type
  Collision {.pure.} = enum x, y, corner

const
  playerSize = vector2d(64, 64)

  air = 0
  start = 78
  finish = 110

getTile reads a tile from a specified position of the map, making sure not to over- or underflow:

proc getTile(map: Map, x, y: int): uint8 =
  let
    nx = clamp(x div tileSize.x, 0, map.width - 1)
    ny = clamp(y div tileSize.y, 0, map.height - 1)
    pos = ny * map.width + nx

  map.tiles[pos]

isSolid determines whether the player can collide with a tile. As we said every tile other than air, start and finish are able to be collided with:

proc isSolid(map: Map, x, y: int): bool =
  map.getTile(x, y) notin {air, start, finish}

proc isSolid(map: Map, point: Point2d): bool =
  map.isSolid(point.x.round.int, point.y.round.int)

A player is on the ground when there is a block below either side of its feet:

proc onGround(map: Map, pos: Point2d, size: Vector2d): bool =
  let size = size * 0.5
  result =
    map.isSolid(point2d(pos.x - size.x, pos.y + size.y + 1)) or
    map.isSolid(point2d(pos.x + size.x, pos.y + size.y + 1))

Meanwhile testBox considers the player as an axis aligned boundary box, and tells us if the player is stuck inside of any solid walls:

proc testBox(map: Map, pos: Point2d, size: Vector2d): bool =
  let size = size * 0.5
  result =
    map.isSolid(point2d(pos.x - size.x, pos.y - size.y)) or
    map.isSolid(point2d(pos.x + size.x, pos.y - size.y)) or
    map.isSolid(point2d(pos.x - size.x, pos.y + size.y)) or
    map.isSolid(point2d(pos.x + size.x, pos.y + size.y))

moveBox now tries to apply the velocity vector vel to the player’s position pos. When this causes a collision the code tries to move the player only along the x axis, then only along the y axis, to find out which side of the tile the player collided with. If the player did not collide with any of the sides it hit a corner (that’s the real corner case!):

proc moveBox(map: Map, pos: var Point2d, vel: var Vector2d,
             size: Vector2d): set[Collision] {.discardable.} =
  let distance = vel.len
  let maximum = distance.int

  if distance < 0:
    return

  let fraction = 1.0 / float(maximum + 1)

  for i in 0 .. maximum:
    var newPos = pos + vel * fraction

    if map.testBox(newPos, size):
      var hit = false

      if map.testBox(point2d(pos.x, newPos.y), size):
        result.incl Collision.y
        newPos.y = pos.y
        vel.y = 0
        hit = true

      if map.testBox(point2d(newPos.x, pos.y), size):
        result.incl Collision.x
        newPos.x = pos.x
        vel.x = 0
        hit = true

      if not hit:
        result.incl Collision.corner
        newPos = pos
        vel = vector2d(0, 0)

    pos = newPos

The moveBox proc also returns a set of collisions, telling us what kind of collisions happened in this iteration. We don’t use that information, but it could be useful if you want to handle collisions in a special way instead of just pushing the player out of the collided wall.

Finally we can use onGround and moveBox:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  let ground = game.map.onGround(game.player.pos, playerSize)

  if game.inputs[Input.jump]:
    if ground:
      game.player.vel.y = -21

  let direction = float(game.inputs[Input.right].int -
                        game.inputs[Input.left].int)

  game.player.vel.y += 0.75
  if ground:
    game.player.vel.x = 0.5 * game.player.vel.x + 4.0 * direction
  else:
    game.player.vel.x = 0.95 * game.player.vel.x + 2.0 * direction
  game.player.vel.x = clamp(game.player.vel.x, -8, 8)

  game.map.moveBox(game.player.pos, game.player.vel, playerSize)

Jumping is only possible when standing on the ground. Horizontal movement in the air is calculated in a different way than on the ground, simulating different air and ground friction.

Full code for section 5

6. Camera

Did you see me disappear out of the window? Nice, right?! Well, actually that probably makes playing the rest of the map rather difficult. So while we can move nicely, the camera’s position is still fixed. In this section we will only make the camera move horizontally. With this information you can probably figure out how to make it move vertically as well if you want to. We only need to set the camera position when our player moved, right after calling game.physics():

const
  windowSize: Point = (1280.cint, 720.cint)

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  game.camera.x = game.player.pos.x - halfWin

for tick in lastTick+1 .. newTick:
  game.physics()
  game.moveCamera()

Now the camera always immediately follows the player’s horizontal position. Another approach for the camera is to have it follow the player only once the player leaves the center area of the screen, in this case 200 pixels:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  let
    leftArea  = game.player.pos.x - halfWin - 100
    rightArea = game.player.pos.x - halfWin + 100
  game.camera.x = clamp(game.camera.x, leftArea, rightArea)

An alternative approach is to have the camera follow the player fluidly. When the player is further away from the center the camera moves towards him faster. You can imagine the camera as being pulled by a rubber band connecting it to the player, the further away they get, the stronger the camera gets pulled to the player:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  let dist = game.camera.x - game.player.pos.x + halfWin
  game.camera.x -= 0.05 * dist

Choices are difficult so we can just implement all three and you can choose at compile time using -d:fluidCamera and -d:innerCamera:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  when defined(fluidCamera):
    let dist = game.camera.x - game.player.pos.x + halfWin
    game.camera.x -= 0.05 * dist
  elif defined(innerCamera):
    let
      leftArea  = game.player.pos.x - halfWin - 100
      rightArea = game.player.pos.x - halfWin + 100
    game.camera.x = clamp(game.camera.x, leftArea, rightArea)
  else:
    game.camera.x = game.player.pos.x - halfWin

Full code for section 6

7. Game State

Now that we can run around all of the map with the camera keeping up with us, we should give our game a purpose. You might have noticed the light and dark gray lines at the beginning and end of the map, suspiciously referred to as start and finish respectively. As you can probably guess we will use those as start and finish lines and record how quickly the player can get from start to finish:

type
  Time = ref object
    begin, finish, best: int

  Player = ref object
    texture: TexturePtr
    pos: Point2d
    vel: Vector2d
    time: Time

proc restartPlayer(player: Player) =
  player.pos = point2d(170, 500)
  player.vel = vector2d(0, 0)
  player.time.begin = -1
  player.time.finish = -1

proc newTime: Time =
  new result
  result.finish = -1
  result.best = -1

proc newPlayer(texture: TexturePtr): Player =
  new result
  result.texture = texture
  result.time = newTime()
  result.restartPlayer()

We’re now storing a Time object in the Player, telling us when the player began playing this round, how he finished last time and what his absolute best time is. By default the values are initialized to -1 to indicate an invalid value, otherwise they store ticks.

To format the time for display we use the excellent strfmt library’s string interpolation:

import strfmt

proc formatTime(ticks: int): string =
  let
    mins = (ticks div 50) div 60
    secs = (ticks div 50) mod 60
    cents = (ticks mod 50) * 2
  interp"${mins:02}:${secs:02}:${cents:02}"

Our game logic works as follows:

When the player walks through a start tile his time begins
When the player walks through a finish tile his finish time is set and printed to the terminal. If it is a new best time, so is his best time. The start time is reset.

proc getTile(map: Map, pos: Point2d): uint8 =
  map.getTile(pos.x.round.int, pos.y.round.int)

proc logic(game: Game, tick: int) =
  template time: untyped = game.player.time
  case game.map.getTile(game.player.pos)
  of start:
    time.begin = tick
  of finish:
    if time.begin >= 0:
      time.finish = tick - time.begin
      time.begin = -1
      if time.best < 0 or time.finish < time.best:
        time.best = time.finish
      echo "Finished in ", formatTime(time.finish)
  else: discard

We need to call the logic in our main proc of course:

for tick in lastTick+1 .. newTick:
  game.physics()
  game.moveCamera()
  game.logic(tick)

Now we can play through the map and finally get an output like this on the terminal:

Finished in 00:04:38

Full code for section 7

8. Text Rendering

It would be much nicer to have the text output with the result in the actual game window instead of on the terminal. For this we will now use SDL_ttf:

import sdl2/ttf

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y: cint, color: Color) =
  let surface = font.renderUtf8Solid(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  var source = rect(0, 0, surface.w, surface.h)
  var dest = rect(x, y, surface.w, surface.h)
  let texture = renderer.createTextureFromSurface(surface)

  sdlFailIf texture.isNil:
    "Could not create texture from rendered text"

  surface.freeSurface()

  renderer.copyEx(texture, source, dest, angle = 0.0, center = nil,
                  flip = SDL_FLIP_NONE)

  texture.destroy()

proc renderText(game: Game, text: string,
                x, y: cint, color: Color) =
  const outlineColor = color(0, 0, 0, 64)
  game.renderer.renderText(game.font, text, x, y, color)

What’s happening here is that we render a text with a FontPtr to an SDL2 surface (stored in RAM), which is then put into a texture (stored in GPU’s VRAM). This texture is then rendered to the screen at the defined position.

proc render(game: Game, tick: int) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)

  let time = game.player.time
  const white = color(255, 255, 255, 255)
  if time.begin >= 0:
    game.renderText(formatTime(tick - time.begin), 50, 100, white)
  elif time.finish >= 0:
    game.renderText("Finished in: " & formatTime(time.finish),
      50, 100, white)
  if time.best >= 0:
    game.renderText("Best time: " & formatTime(time.best),
      50, 150, white)

  # Show the result on screen
  game.renderer.present()

We need to initialize the TTF subsystem and now we also need to pass the current tick to render inside of our main function:

sdlFailIf(ttfInit() == SdlError): "SDL2 TTF initialization failed"
defer: ttfQuit()

...

game.render(lastTick)

As well as loading our DejaVuSans.ttf font inside of newGame:

result.font = openFont("DejaVuSans.ttf", 28)
sdlFailIf result.font.isNil: "Failed to load font"

The Game object now looks like this, including a font:

type
  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    font: FontPtr
    player: Player
    map: Map
    camera: Vector2d

This is what the text rendering looks like after finishing the map:

It noticeably doesn’t look all that great. The border of the text looks rugged. This happens because renderUtf8Solid does not use alpha blending, which is expensive. Instead every pixel is either entirely white or entirely transparent, never anything half-transparent in between. If we had a fixed background color for the text we could use renderUtf8Shaded, which takes a background color. If we want nicer output with dynamic backgrounds we can use renderUtf8Blended instead:

This looks better, but would be hard to see with a brighter background. We can draw an outline for our text to fix this, basically by drawing the text twice, once in half-transparent black and once in the proper color on top:

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y, outline: cint, color: Color) =
  font.setFontOutline(outline)
  let surface = font.renderUtf8Blended(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  var source = rect(0, 0, surface.w, surface.h)
  var dest = rect(x - outline, y - outline, surface.w, surface.h)
  ...

proc renderText(game: Game, text: string,
                x, y: cint, color: Color) =
  const outlineColor = color(0, 0, 0, 64)
  game.renderer.renderText(game.font, text, x, y, 2, outlineColor)
  game.renderer.renderText(game.font, text, x, y, 0, color)

Full code for section 8

9. Text Caching

If you looked at your CPU usage during this tutorial so far you might have noticed that the game needed nearly no CPU at all, about 3 % for 60 fps on my system. Once the texts are rendered this increases to 20% though. The reason for this is that right now we are regenerating the text textures every single frame, even if it didn’t change from the last frame. If you have fixed texts you can simply save the textures instead of recalculating them. But if you want more flexibility instead you can use a glyph or texture caching system. An example of such a system would be SDL_FontCache.

But instead we can write a little application-specific caching scheme in Nim. The heuristic we use is that mostly a single line in the code base will keep producing the same string, at least for some time. So we cache only a single text rendering for each line that prints something to the screen. That means we don’t have to do any lookups in a cache data structure and we only use a guaranteed constant amount of memory for caching:

type
  CacheLine = object
    texture: TexturePtr
    w, h: cint

  TextCache = ref object
    text: string
    cache: array[2, CacheLine]

proc newTextCache: TextCache =
  new result

A CacheLine is what we store, a pointer to a texture as well as the texture’s width and height. For our text rendering we need two of those as we render the text twice to get the outline effect. The text is also stored in TextCache to see if we already have the correct textures cached.

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y, outline: cint, color: Color): CacheLine =
  font.setFontOutline(outline)
  let surface = font.renderUtf8Blended(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  result.w = surface.w
  result.h = surface.h
  result.texture = renderer.createTextureFromSurface(surface)
  sdlFailIf result.texture.isNil: "Could not create texture from rendered text"

  surface.freeSurface()

We manipulated renderText to return a CacheLine that we can use:

proc renderText(game: Game, text: string, x, y: cint,
                color: Color, tc: TextCache) =
  let passes = [(color: color(0, 0, 0, 64), outline: 2.cint),
                (color: color, outline: 0.cint)]

  if text != tc.text:
    for i in 0..1:
      tc.cache[i].texture.destroy()
      tc.cache[i] = game.renderer.renderText(
        game.font, text, x, y, passes[i].outline, passes[i].color)
    tc.text = text

  for i in 0..1:
    var source = rect(0, 0, tc.cache[i].w, tc.cache[i].h)
    var dest = rect(x - passes[i].outline, y - passes[i].outline,
                    tc.cache[i].w, tc.cache[i].h)
    game.renderer.copyEx(tc.cache[i].texture, source, dest,
                         angle = 0.0, center = nil)

In two passes the text is rendered, if the text is cached from the cache directly, otherwise the old cache entry is removed and replaced with the new textures.

Nim’s metaprogramming allows us to use this seamlessly with a small template. A few days ago I wrote an article about Metaprogramming in Nim if you want to learn more about the powerful side of Nim.

template renderTextCached(game: Game, text: string,
                          x, y: cint, color: Color) =
  block:
    var tc {.global.} = newTextCache()
    game.renderText(text, x, y, color, tc)

proc render(game: Game, tick: int) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)

  let time = game.player.time
  const white = color(255, 255, 255, 255)
  if time.begin >= 0:
    game.renderTextCached(formatTime(tick - time.begin),
      50, 100, white)
  elif time.finish >= 0:
    game.renderTextCached("Finished in: " &
      formatTimeExact(time.finish), 50, 100, white)
  if time.best >= 0:
    game.renderTextCached("Best time: " &
      formatTimeExact(time.best), 50, 150, white)

  # Show the result on screen
  game.renderer.present()

Now each of the three renderTextCached calls get its own TextCache assigned, which is then used for the rest of the program execution. Note that this caching scheme only works under the assumption that there are relatively few separate lines of code that call renderTextCached and the ones that do often render the same text multiple times in a row. Good enough for our use case.

proc formatTime(ticks: int): string =
  let mins = (ticks div 50) div 60
  let secs = (ticks div 50) mod 60
  interp"${mins:02}:${secs:02}"

proc formatTimeExact(ticks: int): string =
  let cents = (ticks mod 50) * 2
  interp"${formatTime(ticks)}:${cents:02}"

We also reduced the exactness of the current time format because before it had to be recalculated every single frame. Now we’re back to using 4 % CPU and our final game looks like this:

Full code for section 9

Building

We can create a platformer.nimble file to tell Nimble, Nim’s package manager, how to build our package:

# Package

version       = "1.0"
author        = "Dennis Felsing"
description   = "An example platform game with SDL2"
license       = "MIT"

bin           = @["platformer"]

# Dependencies

requires "nim >= 0.10.0"
requires "sdl2 >= 1.1"
requires "strfmt >= 0.6"
requires "basic2d >= 0.1.0"

task tests, "Compile all tutorial steps":
  for i in 1..9:
    exec "nim c tutorial/platformer_part" & $i

All the intermediate code states from this article can be compiled with nimble tests. We can test the build by running nimble build, which creates a platformer binary that we can run. nimble install installs the same binary so that it is available in ~/.nimble/bin.

But once we run this installed platformer binary we get an error:

Error: unhandled exception: Could not load image player.png, SDL error: Couldn't open player.png [SDLException]

Actually this makes sense, because we load the files at runtime from the current directory, which can now be any directory. We have two choices for how to solve this:

Read in all files at compile time and get a binary that depends on no assets
Doing this is pretty simple in Nim as the compiler supports reading arbitrary files at compile time with staticRead:

import streams

template staticReadRW(filename: string): ptr RWops =
  const file = staticRead(filename)
  rwFromConstMem(file.cstring, file.len)

template staticReadStream(filename: string): string =
  const file = staticRead(filename)
  newStringStream(file)

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

  result.font = openFontRW(
    staticReadRW("DejaVuSans.ttf"), freesrc = 1, 28)
  sdlFailIf result.font.isNil: "Failed to load font"

  result.player = newPlayer(renderer.loadTexture_RW(
    staticReadRW("player.png"), freesrc = 1))
  result.map = newMap(renderer.loadTexture_RW(
    staticReadRW("grass.png"), freesrc = 1),
    staticReadStream("default.map"))

Define a directory where we store our data assets to load them at runtime. This enables the player to switch them out for custom ones without the need to recompile the binary. And as we’ve seen with the DDNet Skin Database this might be a useful feature. We find the data directory by looking where in the file system our binary is. Let’s implement this and make the old embedded compile time assets optional:

import os, streams

const dataDir = "data"

when defined(embedData):
  template readRW(filename: string): ptr RWops =
    const file = staticRead(dataDir / filename)
    rwFromConstMem(file.cstring, file.len)

  template readStream(filename: string): Stream =
    const file = staticRead(dataDir / filename)
    newStringStream(file)
else:
  let fullDataDir = getAppDir() / dataDir

  template readRW(filename: string): ptr RWops =
    var rw = rwFromFile(cstring(fullDataDir / filename), "r")
    sdlFailIf rw.isNil: "Cannot create RWops from file"
    rw

  template readStream(filename: string): Stream =
    var stream = newFileStream(fullDataDir / filename)
    if stream.isNil: raise ValueError.newException(
      "Cannot open file stream:" & fullDataDir / filename)
    stream

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

  result.font = openFontRW(
    readRW("DejaVuSans.ttf"), freesrc = 1, 28)
  sdlFailIf result.font.isNil: "Failed to load font"

  result.player = newPlayer(renderer.loadTexture_RW(
    readRW("player.png"), freesrc = 1))
  result.map = newMap(renderer.loadTexture_RW(
    readRW("grass.png"), freesrc = 1),
    readStream("default.map"))

This also requires us to change newMap to accept a Stream instead of a filename:

proc newMap(texture: TexturePtr, map: Stream): Map =
  new result
  result.texture = texture
  result.tiles = @[]

  var line = ""
  while map.readLine(line):
    ...

When compiling you can set -d:embedData like this:

$ nim -d:release c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 129K Jun 13 14:54 platformer*
$ nim -d:release -d:embedData c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 888K Jun 13 14:55 platformer*

You can find our final platform game code in the repository.

Now we can submit the repository to the Nimble packages as a pull request and soon all Nim developers can install the package right from their terminal with a nimble install platformer and play by just running platformer.

A circle.yml file defines how to run and compile our repository and is used to make sure it keeps building fine whenever changes are made.

Binary Distribution

But Nim developers are probably not the main target group of our game, so ideally we we also want to be able to build binaries for a few common platforms, Linux x86, x86-64 and Windows x86, x86-64 for us. Building for Mac OS X is a bit more involved, but you can check out how DDNet does it.

Of course we could just set up a VM for each system that we want to build for and use the instructions from the start of this article. But that’s tedious and we want the convenience of building on a single machine.

Linux

Note that I’m on Arch Linux, but this should be possible in an analogous way on other Linux distributions:

Building a portable binary for Linux is a pain because of glibc. When you compile on a system with a newer glibc version it might not run on a system with an older one. A common solution is to use a build system with an old Linux install. Alternatively an old Debian chroot can be created with debootstrap. There is also Linux Standard Base which aims to solve this problem, but I haven’t used yet.

A more hacky solution is to create the binary on the new system and check what symbols exactly are linked against the newer glibc version. In our case I want everything to use GLIBC_2.2.5, so I check for anything else:

objdump -T platformer | grep GLIBC | grep -v 2.2.5
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy

Okay, only memcpy is the problem. We can force the linker to use an old version of memcpy and another common problem, realpath, like this in C code with inline assembler:

__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");

But this would require for us to insert this into every C file that we generate. Or we abuse the nimbase.h file and insert it there and compile with:

$ head -n2 glibc-hack/nimbase.h
__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");
$ nim -d:release --passC:-Iglibc-hack c platformer
$ objdump -T platformer | grep memcpy
0000000000000000      DF *UND* 0000000000000000  GLIBC_2.2.5 memcpy
$ objdump -T platformer|grep GLIBC|grep -v 2.2.5

Now our binary works on Linux versions using glibc 2.2.5 or newer. Note that the user still needs SDL2, SDL_image2 and SDL_ttf2 installed.

When you’re linking dynamically and want to distribute shared libraries with your binary you can compile with nim --passC:-Wl,-rpath,. c platformer and put the shared libraries into the same directory as the binary.

At least building for x86 is easy on Linux as long as you’re on x86-64 and have gcc-multilib installed:

$ yaourt -S lib32-sdl2 lib32-sdl2_image lib32-sdl2_ttf
$ nim --cpu:i386 --passC:-m32 --passL:-m32 -d:release c platformer

Windows

Surprisingly it is much easier to build portable binaries for Windows, even from Linux:

$ pacman -S mingw-w64-gcc
$ nim --os:windows --cpu:amd64 --gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc -d:release c platformer
$ nim --os:windows --cpu:i386 --gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc -d:release c platformer

SDL libraries for Windows can be downloaded from the SDL2 website (image, ttf).

Stripping the binaries of symbols with strip -s platformer is a good idea if you want to save some space and don’t care about debugging your binary.

Automated Build Script

With all this information we can now write a fully automated release build script, written in Nim as well:

import os, strfmt

const
  app = "platformer"
  version = "1.0"

  builds = [
    (name: "linux_x86", os: "linux", cpu: "i386",
     args: "--passC:-m32 --passL:-m32"),
    (name: "linux_x86_64", os: "linux", cpu: "amd64",
     args: "--passC:-Iglibc-hack"),
    (name: "win32", os: "windows", cpu: "i386",
     args: "--gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc"),
    (name: "win64", os: "windows", cpu: "amd64",
     args: "--gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc"),
  ]

removeDir "builds"

for name, os, cpu, args in builds.items:
  let
    dirName = app & "_" & version & "_" & name
    dir = "builds" / dirName
    exeExt = if os == "windows": ".exe" else: ""
    bin = dir / app & exeExt

  createDir dir
  if execShellCmd(interp"nim --cpu:${cpu} --os:${os} ${args} -d:release -o:${bin} c ${app}") != 0: quit 1
  if execShellCmd(interp"strip -s ${bin}") != 0: quit 1
  copyDir("data", dir / "data")
  if os == "windows": copyDir("libs" / name, dir)
  setCurrentDir "builds"
  if os == "windows":
    if execShellCmd(interp"zip -9r ${dirName}.zip ${dirName}") != 0: quit 1
  else:
    if execShellCmd(interp"tar cfz ${dirName}.tar.gz ${dirName}") != 0: quit 1
  setCurrentDir ".."

Linux users have to install sdl2, sdl2_image, sdl2_ttf using their package manager. Windows users get them bundled. Our build script creates this directory structure when we run it with nim -r c release:

.
├── platformer_1.0_linux_x86
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   └── platformer
├── platformer_1.0_linux_x86.tar.gz
├── platformer_1.0_linux_x86_64
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   └── platformer
├── platformer_1.0_linux_x86_64.tar.gz
├── platformer_1.0_win32
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   ├── libfreetype-6.dll
│   ├── libpng16-16.dll
│   ├── platformer.exe
│   ├── SDL2.dll
│   ├── SDL2_image.dll
│   ├── SDL2_ttf.dll
│   └── zlib1.dll
├── platformer_1.0_win32.zip
├── platformer_1.0_win64
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   ├── libfreetype-6.dll
│   ├── libpng16-16.dll
│   ├── platformer.exe
│   ├── SDL2.dll
│   ├── SDL2_image.dll
│   ├── SDL2_ttf.dll
│   └── zlib1.dll
└── platformer_1.0_win64.zip

Resulting downloads here: Win64, Win32, Linux x86_64, Linux x86

Final Words

Somehow my articles are getting longer and this journey towards writing a simple platform game took a few more explanations than expected. I hope this article wasn’t too long and was instructive and helpful to get an understanding of how to write platform games in Nim with SDL2.

All the material for this article is available in the repository on GitHub. Because I changed around things late into the article I might have made a mistake or missed something. If you find a bug or have a comment you can drop me an email at dennis@felsing.org.

Discussions on r/programming and Hacker News.

Introduction to Metaprogramming in Nim

2016-06-06T00:00:00-04:00

Introduction to the Introduction (Meta-Introduction)

Wikipedia gives us a nice description of metaprogramming:

Metaprogramming is the writing of computer programs with the ability to treat programs as their data. It means that a program could be designed to read, generate, analyse and/or transform other programs, and even modify itself while running.

In this article we will explore Nim’s metaprogramming capabilities, which are quite powerful and yet still easy to use. After all great metaprogramming is one of Nim’s main features. The general rule is to use the least powerful construct that is still powerful enough to solve a problem, in this order:

Normal procs and inline iterators
Generic procs and closure iterators
Templates
Macros

So before looking at Nim’s two main metaprogramming constructs, templates and macros, we’ll look at what we can do with procs and iterators as well.

Regular Programming Constructs

Normal procs

We’re in normal programming land here. Regular procedures are what you know as functions elsewhere and they’re pretty easy to define and use:

proc sayHi(name: string) =
  echo "Hello ", name

sayHi("World")
sayHi "World"
"World".sayHi

Generic procs

With generics we can define procs that work on multiple types. Actually a new proc will be generated based on our generic definition for each instantiation:

proc min[T](x, y: T): T =
  if x < y: x else: y

echo min(2, 3) # more explicitly: min[int](2, 3)
echo min("foo", "bar") # min[string]("foo", "bar")

Inline iterators

Inline iterators are the default iterators in Nim. They get compiled into high performance loops:

iterator reverseItems(x: string): char =
  for i in countdown(x.high, x.low):
    yield x[i]

for c in "foo".reverseItems:
  echo c

So this code gets compiled into:

let x = "foo"
for c in countdown(x.high, x.low):
  let c = x[i]
  echo c

Of course we can make iterators generic too:

iterator reverseItems[T](x: T): auto =
  for i in countdown(x.high, x.low):
    yield x[i]

Closure iterators

Inline iterators simultaneously have the advantage and disadvantage of being translated into loops. This means you can not pass them around. This limitation can be lifted by using closure iterators instead:

import math

proc powers(m: int): auto =
  #return iterator: int {.closure.} = # Make a closure explicitly
  return iterator: int = # Compiler makes this a closure for us
    for n in 0..int.high:
      yield n^m

var
  squares = powers(2)
  cubes = powers(3)

for i in 1..4:
  echo "Square: ", squares() # 0, 1, 4, 9
for i in 1..4:
  echo "Cube: ", cubes()     # 0, 1, 8, 27
echo "Square: ", squares()   # 16
echo "Cube: ", cubes()       # 64

for x in squares(): # Go through all the remaining squares
  echo "Square: ", x         # 25, 36, 49, 64, ...

As you can see closure iterators keep their state. You can call them again and get the next value, or use them inside of a for-loop to get out many values.

Templates

You can think of templates as Nim’s equivalent to the C preprocessor. But templates are written in Nim itself and fit well into the rest of the language.

Templates simply insert their code at the invocation site, working at the level of the abstract syntax tree. They can be used in just the same way as procs.

Logger

A common example are loggers, which we looked at in another article already. Consider that you want to have extensive debug logging in your program. A trivial implementation would look like this:

import strutils, times, os

type Level* {.pure.} = enum
  debug, info, warn, error, fatal

var logLevel* = Level.debug

proc debug*(args: varargs[string, `$`]) =
  if logLevel <= Level.debug:
    echo "[$# $#]: $#" % [getDateStr(), getClockStr(),
      join args]

proc expensiveDebuggingInfo*: string =
  sleep(milsecs = 1000)
  result = "Everything looking good!"

debug expensiveDebuggingInfo()

[2016-06-05 22:00:50]: Everything looking good!

We have to call expensiveDebuggingInfo to get the debugging info, which is fine right now since our logLevel is set to Level.debug. But it stops being fine when we instead set logLevel to anything higher than debug. Then it still takes a full second to evaluate the expensiveDebuggingInfo parameter for debug, but inside of debug nothing is done with that information. This is of course a consequence of call-by-value argument evaluation, which Nim uses, just as most other languages do. A notable exception would be lazy evaluation in Haskell, where this kind of logger would work perfectly fine, only calling expensiveDebuggingInfo when its value is actually needed.

But let’s stay in Nim-land and use a template instead of a proc to magically fix this:

template debug*(args: varargs[string, `$`]) =
  if logLevel <= Level.debug:
    const module = instantiationInfo().filename[0 .. ^5]
    echo "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
      module, join args]

[2016-06-05 22:01:30][logger]: Everything looking good!

Note that we also conveniently use instantiationInfo() to find out at what location in the program our template was instantiated, something we could not do using a procedure.

We can still call the template in the exact same way as the proc. But now we have the advantage that the template is inlined at compiletime, so expensiveDebuggingInfo is only called if the runtime logLevel actually requires it. Perfect.

Safe locking

Another problem that can be solved with a template is automatically acquiring and releasing a system lock:

import locks

template withLock(lock: Lock, body: untyped) =
  acquire lock
  try:
    body
  finally:
    release lock

Compile with --threads:on for platform independent lock support.

This looks pretty simple, we just acquire the lock, execute the passed statements and finally release the lock, even if exceptions have been thrown. We can pass any set of statements as the body. The usage is as easy as using a built-in if statement:

var lock: Lock
initLock lock

withLock lock:
  echo "Do something that requires locking"
  echo "This might throw an exception"

When our template accepts a value of type stmt we can use the colon to pass an entire indented block of code. When we have multiple parameters of type stmt the do notation can be used.

This gets transformed into:

var lock: Lock
initLock lock

acquire lock
try:
  echo "Do something that requires locking"
  echo "This might throw an exception"
finally:
  release lock

Now we will never forget to call release lock. You could use this to make a higher level locking library that only exposes withLock instead of the lower-level acquire and release primitives.

Macros

Just like templates, macros are executed at compiletime. But with templates you can only do constant substitutions in the AST. With macros you can analyze the passed arguments and create a new AST at the current position in any way you want. A nice property of Nim is that these compiletime macros are also written in the regular Nim language, so there is no need to learn another language.

A simple way to create an AST is to use parseStmt and parseExpr to parse the regular textual representation into a NimNode. For example parseStmt("result = 10") returns this AST:

StmtList
  Asgn
    Ident !"result"
    IntLit 10

A very useful way to find the AST of a piece of code is dumpTree:

import macros

dumpTree:
  result = 10

This is the same output as you get with treeRepr:

import macros

static:
  echo treeRepr(parseStmt("result = 10"))

Alternatively you can use lispRepr to get a lisp-like representation:

StmtList(Asgn(Ident(!"result"), IntLit(10)))

Finally there is also the repr proc, which turns a NimNode AST back into its textual representation.

Many beginners start by piecing strings together and finally calling parseStmt on them. While this works it is inefficient and prone to bugs. Instead you can use the macros module to create NimNodes of all kinds yourself. dumpTree gives you a hint if you’re not sure how a specific piece of code will look in its AST representation.

JSON Parsing

JSON is pretty popular, so let’s improve the support for it in Nim. What we want is to have a magical %* so that we can write JSON directly in Nim source code and have it checked at compile time, like this:

var j1 = %*
  [
    {
      "name": "John",
      "age": 30
    },
    {
      "name": "Susan",
      "age": 31
    }
  ]

So far if you want to use JSON in Nim, you have to use the JSON constructor % a lot:

import json

var j2 =
  %[
    %{
      "name": %"John",
      "age": %30
    },
    %{
      "name": %"Susan",
      "age": %31
    }
  ]

Looks annoying. How can we implement %*? As a macro of course!:

macro `%*`*(x: expr): expr = toJson(x)

Ok, that doesn’t do anything interesting yet. We just call the still unspecified compile time proc toJson and return the result. We want toJson to traverse the passed AST x and create a new AST, which inserts a % call at just the right places, exactly as it would happen if we added the % calls manually.

For this purpose we print the AST of j2 by putting it into dumpTree from the macros module:

import json, macros

dumpTree:
  %[
    %{
      "name": %"John",
      "age": %30
    },
    %{
      "name": %"Susan",
      "age": %31
    }
  ]

We get the following AST printed when compiling this program:

Prefix
  Ident !"%"
  Bracket
    Prefix
      Ident !"%"
      TableConstr
        ExprColonExpr
          StrLit name
          Prefix
            Ident !"%"
            StrLit John
        ExprColonExpr
          StrLit age
          Prefix
            Ident !"%"
            IntLit 30
    Prefix
      Ident !"%"
      TableConstr
        ExprColonExpr
          StrLit name
          Prefix
            Ident !"%"
            StrLit Susan
        ExprColonExpr
          StrLit age
          Prefix
            Ident !"%"
            IntLit 31

This turned out quite big, but from here we can see how the AST we want to construct looks like. We do the same for j1 to see what we’re working with:

StmtList
  Bracket
    TableConstr
      ExprColonExpr
        StrLit name
        StrLit John
      ExprColonExpr
        StrLit age
        IntLit 30
    TableConstr
      ExprColonExpr
        StrLit name
        StrLit Susan
      ExprColonExpr
        StrLit age
        IntLit 31

The idea now is to insert a % at each level, except in front of the "name" and "age" in our case, the first elements in colon expressions.

proc toJson(x: PNimrodNode): PNimrodNode {.compiletime.} =
  case x.kind
  of nnkBracket: # Corresponds to Bracket in dumpTree
    result = newNimNode(nnkBracket)
    for i in 0 .. <x.len:
      result.add(toJson(x[i])) # Recurse to add %

  of nnkTableConstr: # nnk stands for Nim node kind
    result = newNimNode(nnkTableConstr)
    for i in 0 .. <x.len:
      assert x[i].kind == nnkExprColonExpr
      result.add(newNimNode(nnkExprColonExpr)
        .add(x[i][0])           # First element: no %
        .add(toJson(x[i][1]))): # Second element: Recurse to add %

  else:
    result = x # End of recursion

  result = result.prefix("%") # Surround this level with %

And that’s it! Now our %* works just as we want it to. If we did anything wrong, we can modify the macro to check the actual code it produces:

macro `%*`*(x: expr): expr =
  result = toJson(x)
  echo result.repr # Print code representation of AST

This prints:

% [% {"name": % "John", "age": % 30}, % {"name": % "Susan", "age": % 31}]

Perfect! This macro we just developed landed in Nim’s json module already.

Enum Parsing optimization

With enums we can create new types that contain ordered values, just like this:

  type Fruit = enum Apple, Banana, Cherry

Strings can be parsed to an enum using parseEnum from strutils:

  let fruit = parseEnum[Fruit]("cherry")

If we do this a lot, we notice that it’s kind of slow though:

for i in 1 .. 10_000_000:
  var select = parseEnum[Fruit]("cherry")
  doAssert select == Cherry

This takes 2.2 seconds on my machine. Let’s look at the definition of parseEnum to find out why:

proc parseEnum*[T: enum](s: string): T =
  ## Parses an enum ``T``.
  ##
  ## Raises ``ValueError`` for an invalid value in `s`. The
  ## comparison is done in a style insensitive way.
  for e in low(T)..high(T):
    if cmpIgnoreStyle(s, $e) == 0:
      return e
  raise newException(ValueError, "invalid enum value: " & s)

We can see the problem already. We iterate through all the values inside the enum type, from low(T) to high(T). Then $e creates a string of each enum value, which is quite expensive. Since we already know the type of the enum at compile time, we could create the strings at compile time as well.

Again, let’s think about what we want the result to look like before writing the macro. Basically what we want to do is unroll the for loop at compile time:

if cmpIgnoreStyle(s, "Apple") == 0: return Apple
if cmpIgnoreStyle(s, "Banana") == 0: return Banana
if cmpIgnoreStyle(s, "Cherry") == 0: return Cherry
raise newException(ValueError, "invalid enum value: " & s)

Now we can create the proc. Other than in the last example we won’t create the AST manually this time. Instead we use parseStmt to create a statement AST from a string containing Nim code. An equivalent parseExpr for expressions exists as well. Here’s how the final proc with a macro inside looks:

proc parseEnum*[T: enum](s: string): T =
  macro m: untyped =
    result = newStmtList()
    for e in T: result.add parseStmt(
      "if cmpIgnoreStyle(s, \"$1\") == 0: return $1".format(e))

    result.add parseStmt(
      "raise newException(ValueError, \"invalid enum value: \"&s)")

    #echo result.repr # To make sure we get what we want

  m() # Actually invoke the macro to insert the statements here

Running the same code with our new implementation of parseEnum takes 0.5 seconds now, about 4 times faster than before. Great!

HTML DSL

We can use Nim’s templates and macros to create domain specific languages (DSL) that are translated into Nim code at compiletime. Nim’s syntax is quite flexible, so this is a powerful tool. As an example we build a simple HTML DSL.

The goal is to be able to write this:

proc page(title, content: string) {.htmlTemplate.} =
  html:
    head:
      title: title
    body:
      h1: title
      p: "Default Content"
      p: content

echo page("My own website", "My extra content")

And thus print the following HTML:

<html>
  <head>
    <title>
      My own website
    </title>
  </head>
  <body>
    <h1>
      My own website
    </h1>
    <p>
      Default Content
    </p>
    <p>
      My extra content
    </p>
  </body>
</html>

For convenience we want to use the htmlTemplate macro as a pragma, annotated as {.htmlTemplate.}. Instead we could also write it in this way:

htmlTemplate:
  proc page(title, content: string) =
    html:
      head:
        title: title
      body:
        h1: title
        p: "Default Content"
        p: content

The htmlTemplate macro shall transform the page proc, adding a string return type and creating a new body out of the DSL definition, into this:

proc page(title, content: string): string =
  result = ""
  result.add "<html>\n"
  ...
  result.add "</html>\n"

Looks simple enough, here’s how the macro works:

macro htmlTemplate(procDef: expr): untyped =
  procDef.expectKind nnkProcDef

  # Same name as specified
  let name = procDef[0]

  # Return type: string
  var params = @[newIdentNode("string")]
  # Same parameters as specified
  for i in 1..<procDef[3].len:
    params.add procDef[3][i]

  var body = newStmtList()
  # result = ""
  body.add newAssignment(newIdentNode("result"),
    newStrLitNode(""))
  # Recurse over DSL definition
  body.add htmlInner(procDef[6])

  # Return a new proc
  result = newStmtList(newProc(name, params, body))

The real magic of recursively handling the HTML tags happens in htmlInner of course, a compiletime proc that calls itself recursively to iterate over the body definition:

template write(arg: expr) =
  result.add newCall("add", newIdentNode("result"), arg)

template writeLit(args: varargs[string, `$`]) =
  write newStrLitNode(args.join)

proc htmlInner(x: NimNode, indent = 0): NimNode
              {.compiletime.} =
  x.expectKind nnkStmtList
  result = newStmtList()
  let spaces = repeat(' ', indent)
  for y in x:
    case y.kind
    of nnkCall:
      y.expectLen 2
      let tag = y[0]
      tag.expectKind nnkIdent
      writeLit spaces, "<", tag, ">\n"
      # Recurse over child
      result.add htmlInner(y[1], indent + 2)
      writeLit spaces, "</", tag, ">\n"
    else:
      writeLit spaces
      write y
      writeLit "\n"

We can check that we get the expected output by adding a simple echo result.repr at the end of htmlTemplate:

proc page(title, content: string): string =
  result = ""
  add(result, "<html>\x0A")
  add(result, "  <head>\x0A")
  ...
  add(result, "</html>\x0A")

Where \x0A is just the newline character. Looks good and the output works!

emerald is a much more complete HTML DSL that works in a similar manner. A simpler HTML generator is included in the standard library in the htmlgen module.

Conclusion

I hope you enjoyed this trip through Nim’s metaprogramming capabilities. Always remember: With great power comes great responsibility, so use the least powerful construct that does the job. This reduces complexity and makes it easier to understand the code and keep it maintainable.

For further information and reference see:

Discuss on Hacker News and r/programming.

DDNet Server Statistics with ServerStatus, RRDtool and Nim

2016-05-14T00:00:00-04:00

About a month ago I set up statistics for the official DDNet servers. My motivations for this are:

Monitor the servers more easily
Get notified about server problems
Have nice graphs to look at

The choices for the software used are mainly made to keep resource usage low, a general principle used for DDNet since we run on cheap VPSes all around the world and are limited in CPU and memory resources. In the rest of this post we will explore the 3 major tools used, their purpose in our solution as well as their performance impact:

ServerStatus: Gather live server statistics
RRDtool: Record and graph data
Nim: Favorite programming language for performance and readability

Gathering live server statistics with ServerStatus

We’ve been running BotoX’s ServerStatus to get live server statistics for some time now. It works quite well to quickly notice major server problems like a high load or incoming (D)DoS attacks, provided that you keep an eye out for it.

We use ServerStatus by running its simple Python client on each server to gather interesting information. The client transmits that data by TCP to the C/C++ server, which aggregates it into a JSON file. This JSON file is then fetched and displayed every two seconds by the JavaScript frontend of our Status page.

On a regular Saturday morning the end result looks like this:

On a quick glance you notice that the DDNet.tw server has high CPU usage (which is totally normal since it runs some hefty cron jobs every 20 minutes) and DDNet RUS is receiving a small DoS attack with just 1.6 MB/s (unfortunately also totally normal). Apart from that everything looks fine.

ServerStatus footprint, calculated from Linux /proc statistics as follows (in Nim):

import os, strutils, posix, strfmt
# CPU and memory usage of process, based on PROC(5) and
# http://stackoverflow.com/a/16736599
let
  pid = paramStr(1)
  # Clock ticks per second
  frequency = sysconf(SC_CLK_TCK)
  # Size of memory page in bytes
  pagesize  = sysconf(SC_PAGESIZE)

  uptime    = readFile("/proc/uptime").split[0].parseFloat
  fields    = readFile("/proc/" & pid & "/stat").split

  # Amount of time in user mode
  utime     = fields[13].parseInt
  # Amount of time in kernel mode
  stime     = fields[14].parseInt
  # Amount of children time in user mode
  cutime    = fields[15].parseInt
  # Amount of children time in kernel mode
  cstime    = fields[16].parseInt
  # Time process started after boot
  starttime = fields[21].parseInt

  # Resident Set Size: number of pages in memory
  rssmem    = fields[23].parseInt

  totaltime = utime + stime + cutime + cstime
  seconds   = uptime - (starttime / frequency)

  cpuusage  = 100 * (totaltime / frequency) / seconds
  memusage  = rssmem * pagesize / 1_000_000

echo interp"${cpuusage:.2f} % CPU ${memusage:.2f} MB Memory"

Part	CPU	Memory
Client	0.14 %	3.70 MB
Server (9 clients)	0.58 %	0.18 MB

Recording and graphing data with RRDtool

I haven’t used RRDtool for about 7 years, but it’s still an excellent tool to record data into a fixed-size round robin database. For us three functions of RRDtool are important: create to create the database, update to add a new value to be aggregated into the database, and graph to render the database into a beautiful graph.

CPU, network and memory are the most important resources for me, so their usage should be recorded. Let us use network traffic as an example and create a database:

First we need to think about what data we want to record in the RRD:

sample = 30 seconds

day  =   2880 samples =   6 * 480 pixels, each pixel is 03:00 min
days =  20160 samples =  42 * 480 pixels, each pixel is 21:00 min
days = 141120 samples = 147 * 960 pixels, each pixel is 73:30 min

Then we can use this to create the actual database file:

rrdtool create ddnet.tw-net.rrd `# File name` \
  --step 30 `# Interval in seconds with which data is fed` \
  DS:network_rx:GAUGE:60:0:U `# Data source receiving` \
  DS:network_tx:GAUGE:60:0:U `# DS Sending` \
  RRA:AVERAGE:0.5:6:480 `# Round robin archive for 1 day` \
  RRA:AVERAGE:0.5:42:480 `# RRA for 7 days` \
  RRA:AVERAGE:0.5:147:960 `# RRA for 49 days`

If you’re curious about what exactly happens here, you can find more information in rrdcreate(1).

The resulting ddnet.tw-net.rrd file is just 32 KB in size and will forever stay that exact size. (All our databases together are just 1 MB.) New data in each round robin archive simply overwrites the oldest data. A disadvantage of RRDtool is that you need to think ahead and plan what data you want to store.

The next step is to put new data into our little database, which we should do every 30 seconds:

rrdtool update ddnet.tw-net.rrd N:42:1234

Super simple! 42 is our network_rx value, 1234 the value for network_tx. These values are now aggregated using the AVERAGE and finally put into their respective archives.

Once we have enough values we can finally create the graph, for example for 1 day:

rrdtool graph ddnet.tw-net-1d.png --rigid --base 1000 \
  --width 419 --height 150 --logarithmic --units=si -a PNG \
  `# Calculation over last day only` \
  --vertical-label "Bytes/s" --start now-1d \
  `# Fetch data from RRD file` \
  DEF:network_rx=ddnet.tw-net.rrd:network_rx:AVERAGE \
  DEF:network_tx=ddnet.tw-net.rrd:network_tx:AVERAGE \
  `# Calculate aggregates based on data` \
  VDEF:network_rx_a=network_rx,AVERAGE \
  VDEF:network_rx_m=network_rx,MAXIMUM \
  VDEF:network_rx_c=network_rx,LAST \
  VDEF:network_rx_s=network_rx,TOTAL \
  VDEF:network_tx_a=network_tx,AVERAGE \
  VDEF:network_tx_m=network_tx,MAXIMUM \
  VDEF:network_tx_c=network_tx,LAST \
  VDEF:network_tx_s=network_tx,TOTAL \
  `# Draw area graph in light colors` \
  AREA:network_tx#fee8c8: \
  AREA:network_rx#e0e0e0: \
  `# Draw clear area outline on top` \
  LINE1:network_tx#e34a33:"out" \
  `# Print aggregate values to legend` \
  GPRINT:network_tx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_tx_m:"max\: %6.2lf %sB" \
  GPRINT:network_tx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_tx_s:"sum\: %6.2lf %sB\n" \
  `# Other area outline` \
  LINE1:network_rx#636363:"in " \
  GPRINT:network_rx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_rx_m:"max\: %6.2lf %sB" \
  GPRINT:network_rx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_rx_s:"sum\: %6.2lf %sB\n"

As always, the manual of rrdgraph explains the possibilities.

I’m using RRDtool 1.6.0 instead of 1.4.8 because I very much prefer its density of x-axis labels. Here are the outputs of our database:

RRDtool 1.4.8: RRDtool 1.6.0:

RRDtool footprint:

Part	Runtime	Memory
Create (9 servers)	0.01 s	1.74 MB
Graph (9 servers)	1.99 s	2.82 MB

Putting it together with Nim

To aggregate the raw data from ServerStatus into 30-second packets I use a small Nim program. It automatically creates new databases when a new server is added and keeps them updated:

import common, json, osproc, os, times, strutils, tables

type
  Data = object
    network_rx, network_tx: BiggestInt
    cpu, memory_used, memory_total, swap_used, swap_total: BiggestInt
    load: float

const freq = 30 # report new data to rrd every 30 seconds

var
  lastUpdated: BiggestInt = 0
  dataTable = initTable[string, Data]()
  count = 0
  countTable = initTable[string, int]()

proc rrdCreate(file, dataSources: string) =
  discard execCmd(rrdtool & " create " & file & " --step " & $freq & " " & dataSources &
    " RRA:AVERAGE:0.5:6:480 RRA:AVERAGE:0.5:42:480 RRA:AVERAGE:0.5:147:960")

proc rrdUpdate(file: string, values: varargs[string, `$`]) =
  var valuesString = ""
  for value in values:
    if valuesString.len > 0:
      valuesString.add ":"
    valuesString.add value
  discard execCmd(rrdtool & " update " & file & " N:" & valuesString)

proc updateServer(server: JsonNode) =
  let domain = server["type"].str

  for name, value in dataTable.mgetOrPut(domain, Data()).fieldPairs:
    when value is BiggestInt:
      value += server[name].num
    elif value is float:
      value += server[name].fnum
    else:
      error "Unhandled type in Data object"

  inc countTable.mgetOrPut(domain, 0)

  # Only save data if we got 30 values in the expected time span
  if count == freq and countTable[domain] == freq:
    let data = dataTable[domain]
    if data == Data():
      dataTable.del(domain)
      return
    else:
      dataTable[domain] = Data()

    let
      fileNet = (rrdDir / domain) & "-net.rrd"
      fileCpu = (rrdDir / domain) & "-cpu.rrd"
      fileMem = (rrdDir / domain) & "-mem.rrd"

    if not existsFile fileNet:
      fileNet.rrdCreate("DS:network_rx:GAUGE:60:0:U DS:network_tx:GAUGE:60:0:U")
    if not existsFile fileCpu:
      fileCpu.rrdCreate("DS:cpu:GAUGE:60:0:100 DS:load:GAUGE:60:0:U")
    if not existsFile fileMem:
      filemem.rrdCreate("DS:memory_used:GAUGE:60:0:U DS:memory_total:GAUGE:60:0:U DS:swap_used:GAUGE:60:0:U DS:swap_total:GAUGE:60:0:U")

    fileNet.rrdUpdate(data.network_rx div freq, data.network_tx div freq)
    fileCpu.rrdUpdate(min(data.cpu div freq, 100), data.load / freq)
    fileMem.rrdUpdate(data.memory_used div freq, data.memory_total div freq, data.swap_used div freq, data.swap_total div freq)

proc updateAllServers =
  let statsJson = parseFile statsJsonFile
  let newUpdated = parseBiggestInt statsJson["updated"].str

  if newUpdated <= lastUpdated:
    return

  inc count

  for server in statsJson["servers"]:
    try:
      updateServer(server)
    except:
      discard

  if count == freq:
    count = 0
    for val in countTable.mvalues:
      val = 0

while true:
  let startTime = epochTime()

  updateAllServers()

  sleep(int(epochTime() - startTime + 1) * 1000) # every second

The final graphs can be seen on the DDNet Server Statistics page.

Nim monitor footprint:

Part	CPU	Memory
Monitor (9 servers)	0.03 %	0.89 MB

Alerts through Cron and Mail

Now we certainly have nice graphs, but automated alerts about suspicious events would be even better, for example:

Network traffic over 2 MB/s for 4 min
Memory and swap over 90% for 4 min
CPU over 90% for 21 min
Load over 10 for 21 min
Server unreachable for 1 hour

To check these conditions a cron job is run regularly and thanks to a MAILTO entry a mail is sent when an alert has been triggered.

We can get out a single value from the database to standard output using PRINT:

rrdtool graph x -s-4min \
  DEF:v=ddnet.tw-net.rrd:network_rx:AVERAGE \
  VDEF:vm=v,AVERAGE \
  PRINT:vm:%lf

The alert program itself is written in Nim as well and merely gets a few values from the databases and checks if the limits are exceeded:

import common, osproc, os, json, strutils

proc get(file, value, time: string): float =
  let (output, errorCode) = execCmdEx(rrdtool & " graph x -s -" & time & " DEF:v=" & file & ":" & value & ":AVERAGE VDEF:vm=v,AVERAGE PRINT:vm:%lf")

  if errorCode != 0:
    raise newException(ValueError, "Error code " & $errorCode & " from rrdtool: " & output)

  result = output.splitLines[^2].parseFloat

if paramCount() != 1:
  echo "alert [1d|7d|49d]"
  quit 1

let statsJson = parseFile statsJsonFile
for server in statsJson["servers"]:
  let
    domain = server["type"].str
    name = server["name"].str
    fileNet = (rrdDir / domain) & "-net.rrd"
    fileCpu = (rrdDir / domain) & "-cpu.rrd"
    fileMem = (rrdDir / domain) & "-mem.rrd"

  template alert(s: string) = echo name, " ", s

  case paramStr(1)
  of "1d":
    if fileNet.get("network_rx", "4min") + fileNet.get("network_tx", "4min") > 2_000_000:
      alert "network traffic over 2 MB/s for 4 min"

    if fileMem.get("memory_used", "4min") + fileMem.get("swap_used", "4min") > 0.9 * (fileMem.get("memory_total", "4min") + fileMem.get("swap_total", "4min")):
      alert "memory and swap over 90% for 4 min"

  of "7d":
    if fileCpu.get("cpu", "21min") > 90.0:
      alert "CPU over 90% for 21 min"
    if fileCpu.get("load", "21min") > 10.0:
      alert "Load over 10 for 21 min"

  of "49d":
    let network_rx = fileNet.get("network_rx", "4410")
    if network_rx != network_rx: # NaN
      alert "unreachable for 1 hour"

  else:
    echo "unknown parameter ", paramStr(1)
    quit 1

Nim alert footprint:

Part	Runtime	Memory
Alert (9 servers)	0.25 s	2.12 MB

Conclusion

Live view of the last day of the DDNet.tw web server, also hosting this blog:

You can see the full graphs on the DDNet Server Statistics page. As usual you can find the entire source code in our git repository.

All in all the system runs on very little resources, puts out some nice graphs and alerts me automatically about defined problems.

Part	Runtime	CPU	Memory
Client		0.14 %	3.70 MB
Server (9 clients)		0.58 %	0.18 MB
Create (9 servers)	0.01 s		1.74 MB
Graph (9 servers)	1.99 s		2.82 MB
Monitor (9 servers)		0.03 %	0.89 MB
Alert (9 servers)	0.25 s		2.12 MB

Writing an Async Logger in Nim

2016-01-28T00:00:00-05:00

Surprisingly I’m working on HookRace again. I might share a few interesting code snippets and thoughts in this blog along the way. I’m still going with Nim as the programming language.

For an easy start let’s write a logging module that can be used everywhere in the game’s server as well as client. There are mostly three aspects that I care about:

Can be configured to write to different files and/or stdout
Writes to the disk asynchronously, preventing any blocking when the disk is overloaded
Reasonable performance

Follow along if you want to witness how fun it is to write code in Nim!

Motivation

A few logger implementations for Nim already exist:

logging: A simple logger in the standard library. Configurable, but no async writing
omnilog: An advanced logger with lots of features, no async writing
syslog: A bit too high level for my taste and no Windows support

We could adapt the logging or omnilog modules to our requirements, mainly by adding async writing. For now let’s write our own simple logging module from scratch instead. If it turns out to be usable after a few iterations one might merge it with an existing module. Also, this post would be rather boring if I just ended up using an existing logger.

Implementation

Let’s start with the simplest logger possible:

proc log*(text: string) =
  echo text

log "Hello World!"

We have a procedure log that is exported (*). It takes a value text of type string and prints this text to the standard output. Super simple and can be used like this from another file (or the same one):

import logger
log "Hello World!"

This simply prints Hello World to the terminal for now when we execute it. We want to keep using the logger in this exact format.

Next step: Add a fancy logging format:

import strutils, times

proc log*(text: string) =
  echo "[$# $#]: $#" % [getDateStr(), getClockStr(), text]

This uses the strutils and times modules from Nim’s standard library. The % operator formats the string "[$# $#]: $#" with a date, clock and our text into [2016-01-28 20:04:43]: Hello World!

It would also be nice if we could see which module is actually invoking the logger. The instantiationInfo proc can be used to retrieve the filename and line of a template instatiation at compile time. So let’s make our logger a template, this also improves performance because we don’t even need a function call at runtime anymore:

template log*(text: string) =
  const module = instantiationInfo().filename
  echo "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
    module, text]

We end up with this output:

[2016-01-28 20:14:04][example.nim]: Hello World!

Let’s get rid of the .nim at the end:

  const module = instantiationInfo().filename[0 .. ^5]

[0 .. ^5] gives us the part of the string from position 0 (from the start) to position 5 from the end. New output:

[2016-01-28 20:14:04][example]: Hello World!

Perfect! We’ll keep this exact output format.

Now let’s add the ability to log to multiple files simultaneously. We also just handle stdout as a regular file:

type Logger = object
  file: File

var loggers = newSeq[Logger]()

proc addLogger*(file: File) =
  loggers.add Logger(file: file)

template log*(text: string) =
  let
    module = instantiationInfo().filename[0 .. ^5]
    str = "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
      module, text]
  for logger in loggers:
    logger.file.writeLine str
    logger.file.flushFile

By flushing the file we make sure that the output is actually written to the file right now, not just once the buffer is full. That’s important to us, for example if the program crashes and we’d lose the last few lines of the log otherwise.

Now we have to register a few loggers before we can log something, but that’s the only change:

import logger

addLogger stdout
addLogger open("example.log", fmWrite)

log "Hello World!"

Now we write our log message to stdout as well as a file. If we used the fmAppend file mode instead of fmWrite we could append to the log file, right now we’re overwriting it every time we run our example program.

Here comes our main problem: What happens if we’re logging while the game server is running regularly, but the write to the disk takes a few milliseconds or even longer! The game server gets delayed and our performance is terrible!

Adding a Thread

So let’s add a separate thread to write the log messages. In Nim each thread has its own heap, so we can’t directly pass a string to another thread. Instead we will communicate with our new thread using a channel, similarly to Go channels. We start by defining a Message type to send over our channel:

type
  MessageKind = enum
    write, update, stop

  Message = object
    case kind: MessageKind
    of write:
      module, text: string
    of update:
      loggers: seq[Logger]
    of stop:
      nil

var channel: Channel[Message]

We define three kinds of messages:

write: Write a new log message
update: Update the set of files to log to
stop: Stop the logging thread

Our new proc for logging looks like this:

proc threadLog {.thread.} =
  var loggers = newSeq[Logger]()

  while true:
    let msg = recv channel
    case msg.kind
    of write:
      let str = "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
        msg.module, msg.text]

      for logger in loggers:
        logger.file.writeLine str
        logger.file.flushFile
    of update:
      loggers = msg.loggers
    of stop:
      break

The thread keeps running until it receives a stop message. Calling recv on a channel blocks until a message is received. Now that the main load is moved to a new thread, what do we do in our log template and addLogger proc?

template log*(t: string) =
  const module = instantiationInfo().filename[0 .. ^5]
  channel.send Message(kind: write, module: module, text: t)

proc addLogger*(file: File) =
  loggers.add Logger(file: file)
  channel.send Message(kind: update, loggers: loggers)

Simple enough! We also need to initialize the channel and run the thread:

open channel
var thread: Thread[void]
thread.createThread threadLog

Now we have to compile with nim --threads:on c example to get thread support, or we simply add a nim.cfg that contains threads: on. Finally we run our code again and … nothing happens, no log message is printed.

The problem is that we finish the program too quickly. This kills the thread before it can receive our logging message. So let’s add a proc that manages the shutdown process more gracefully:

proc stopLog {.noconv.} =
  channel.send Message(kind: stop)
  joinThread thread
  close channel

  for logger in loggers:
    if logger.file notin [stdout, stderr]:
      close logger.file

addQuitProc stopLog

That’s it, our logger works! For more convenience we can add log levels and corresponding templates:

type Level* {.pure.} = enum
  debug, info, warn, error, fatal

var logLevel* = Level.info

template debug*(t: string) =
  if logLevel <= Level.debug:
    const module = instantiationInfo().filename[0 .. ^5]
    channel.send Message(kind: write, module: module, text: t)

template info*(t: string) =
  if logLevel <= Level.info:
    const module = instantiationInfo().filename[0 .. ^5]
    channel.send Message(kind: write, module: module, text: t)

...

What if we want to log not just strings? We can instead accept any kind and number of arguments and stringify them automatically using $:

proc send(module: string, args: varargs[string]) =
  var msg = Message(kind: write, module: module, text: "")
  for arg in args:
    msg.text.add arg
  channel.send msg

template log*(args: varargs[string, `$`]) =
  const module = instantiationInfo().filename[0 .. ^5]
  send module, args

Now we can call our logger like this:

var x = 12
info "All ", x, " Bananas"

[2016-01-28 21:57:34][example]: All 12 Bananas

For more advanced formatting one might want to use the excellent strfmt library:

import strfmt
var x = 12
var y = 3.4
info "{} + {:.2f} == {}".fmt(x, y, x.float + y)
info interp"I have $x Bananas!"

[2016-01-28 22:24:01][example]: 12 + 3.40 == 15.4
[2016-01-28 22:24:01][example]: I have 12 Bananas!

Everything works great and we can be sure that hard disk latency will not bite us. But how is the performance?

Optimization

First let’s come up with a simple program that allows us to measure performance:

import logger

addLogger open("bench.log", fmWrite)
for i in 1 .. 1_000_000:
  log "Hello World!"

Build Command	Runtime
`nim c bench` (unoptimized)	31.9 s
`nim -d:release c bench` (optimized)	12.1 s

12 seconds to write a million log messages. I guess that’s good enough for my use case. We’d rather run out of free disk space than have too little performance like this.

Anyway, let’s see if there are any low hanging fruits. We could look at the code and try to isolate some parts to figure out what takes long. Or we compile the benchmark with debugger support using nim -d:release --debugger:native c bench and invoke it with valgrind --tool=callgrind ./bench. Afterwards kcachegrind can help us analyze the function calls:

What stands out is that getDateStr and getClockStr take 33% and 32% respectively of the total time! Since the time string only changes once every second we can cache it:

proc threadLog {.thread.} =
  var
    loggers = newSeq[Logger]()
    lastTime: Time
    timeStr = ""

  while true:
    let msg = recv channel
    case msg.kind
    of write:
      let newTime = getTime()
      if newTime != lastTime:
        timeStr = getLocalTime(newTime).format "yyyy-MM-dd HH:mm:ss"
        lastTime = newTime

      let str = "[$#][$#]: $#" % [timeStr, msg.module, msg.text]

      for logger in loggers:
        logger.file.writeLine str
        logger.file.flushFile
    ...

getTime() gets the time in seconds, so we only format a new timeStr when a new second has arrived.

Code Change	Runtime
Cache time string	4.4 s

Flushing the file after each write still consumes a lot of time. Unfortunately this can’t be seen using valgrind because valgrind’s instrumentation slows down the entire program so much that the flushing itself becomes insignificant.

The trick is to flush the file only when we’re actually done writing, that means when we have no more messages waiting in the channel:

if channel.peek == 0:
  logger.file.flushFile

Code Change	Runtime
Flush only when necessary	2.1 s

A general tip when optimizing Nim code is to prevent string allocations and copying. Instead of using writeLine we can add the newline directly and call write:

let str = "[$#][$#]: $#\n" % [timeStr, msg.module, msg.text]

for logger in loggers:
  logger.file.write str

We can use the same message and simply reset its text string instead of allocating a new one every time:

var msg = Message(kind: write, system: "", text: "")
proc send(moduleName: string, args: varargs[string, `$`]) =
  msg.system = moduleName
  msg.text.setLen(0)
  for arg in args:
    msg.text.add arg
  channel.send msg

These two changes don’t have a huge effect, but it’s better than nothing:

Code Change	Runtime
Prevent allocations	1.8 s

Another idea is to use a non-locking write since we only write from one thread anyway. The POSIX C function fwrite_unlocked can easily be imported and used in Nim:

proc fwriteUnlocked(buf: pointer, size, n: int, f: File): int {.
  importc: "fwrite_unlocked", noDecl.}

proc writeUnlocked(f: File, s: string) =
  if fwriteUnlocked(cstring(s), 1, s.len, f) != s.len:
    raise newException(IOError, "cannot write string to file")

logger.file.writeUnlocked str

Unfortunately this does not have any effect on performance.

So now we can log "Hello World" a million times in 1.8 seconds. For comparison, simply printing "Hello World" a million times, without any formatting or asynchronous writing, takes 2.4 seconds already when redirecting stdout to a file:

for i in 1 .. 1_000_000:
  echo "Hello World!"

This is of course because of the flushing that happens internally in echo. If we opt to use a raw write directly we can write the million strings in just 0.2 s, so there is still headroom for further improvements.

Final words

We implemented a simple logger, saw a few cool Nim features along the way and finally improved the performance by a factor of 6.

An even better optimization would be not to use channels for inter-thread communication. After all even the channels documentation notes that they are slow. Instead we could create a data structue in the heap shared between threads and use this. But I doubt that I will need more performance for logging, so this is good enough for now.

The resulting code of this post and future HookRace code can be found on GitHub.

Porting a NES emulator from Go to Nim

2015-05-01T00:00:00-04:00

Let me get this straight. We have an emulator for 1985 hardware that was written in a pretty new language (Go), ported to a language that isn’t even 1.0 (Nim), compiled to C, then compiled to JavaScript? And the damn thing actually works? That’s kind of amazing.

— Summary by haberman

I spent the last weeks working on NimES, a NES emulator in the Nim programming language. As I really liked fogleman’s NES emulator in Go I ended up mostly porting it to Nim. The source code is so clean that it’s often easier to understand the internals of the NES by reading the source code than by reading documentation about it.

The choice of backend fell on SDL2 for me, contrary to GLFW + PortAudio that the Go version used. This was mainly motivated by the great portability promised by SDL2. Later we will see how porting to JavaScript and Android worked. If you’re impatient and want to play a game, there’s a JS demo.

Comparison of Go and Nim

Most Go concepts are quite trivial to translate to Nim. This made the porting process simple.

Let’s compare some data that I found interesting:

Metric	Go	Nim (clang backend)
Compile command	`go build`	`nimble build`
Fresh compile time	2.1 s¹	1.7 s
Recompile time	1.5 s	0.5 s
Binary size	16 MB (static)	136 KB + 1MB SDL2
Source code size²	3260 lines, 74 KB	2145 lines, 60 KB
CPU usage	71 %	53 %
Memory usage	79 MB	73 MB

¹ Excluding go-glfw, go-gl and portaudio, which take 17 s to compile
² Emulation code only

It’s nice to see Nim doing well. Even the compile time is shorter than that of Go, which is well known for its short compile times. Now that the port seems to be doing fine and should be running on all Desktop platforms, let’s look into some other interesting things we can do with Nim:

JavaScript port via emscripten

Nim has a JavaScript backend, but I don’t trust it to be stable enough for this task yet. So I opted for emscripten instead, which can compile C code into JavaScript. Since Nim outputs C code, this sounds like a perfect fit. Luckily eeeee helped me with getting it started, since he had experience by porting my DDNet client to teewebs.net.

It turned out that emsdk is the easiest way to use emscripten:

$ ./emsdk update
$ ./emsdk install latest
$ ./emsdk activate latest
$ source ./emsdk_env.sh

This may take a while, get a cup of tea. Afterwards we should have the emconfigure, emmake and emcc commands available. We can build regular Nim programs and look at the resulting html file:

$ cat hello.nim
echo "Hello World"
$ nim --cc:clang --clang.exe:emcc --clang.linkerexe:emcc \
  --cpu:i386 -d:release -o:hello.html c hello.nim
$ ls -lha hello.{html,js}
-rw-r--r-- 1 def users 101K Mai  1 19:02 hello.html
-rw-r--r-- 1 def users 385K Mai  1 19:02 hello.js

That’s a pretty cumbersome building command, so we’ll slim it down later. The next step is to build SDL2 for emscripten:

$ hg clone https://hg.libsdl.org/SDL
$ cd SDL
$ emconfigure ./configure --host=asmjs-unknown-emscripten \
  --disable-assembly --disable-threads \
  --enable-cpuinfo=false CFLAGS="-O2"
$ emmake make
$ ls -lha build/.libs/libSDL2.a
-rw-r--r-- 1 def users 1.6M Apr 29 06:58 build/.libs/libSDL2.a

I put the resulting libSDL2.a into the NimES repository under emscripten/ for convenience.

Instead of increasing the cumbersomeness of our build command anymore, NimES’s nim.cfg specifies how to compile when -d:emscripten is set:

@if emscripten:
  define = SDL_Static
  gc = none
  cc = clang
  clang.exe = "emcc"
  clang.linkerexe = "emcc"
  clang.options.linker = ""
  cpu = "i386"
  out = "nimes.html"
  warning[GcMem] = off
  passC = "-Wno-warn-absolute-paths -Iemscripten -s USE_SDL=2"
  passL = "-O3 -Lemscripten -s USE_SDL=2 --preload-file tetris.nes --preload-file pacman.nes --preload-file smb.nes --preload-file smb3.nes -s TOTAL_MEMORY=16777216"
@end

Now a simple nim -d:release -d:emscripten c src/nimes builds the JavaScript port. Note that I’m preloading a few ROMs so that they can be loaded. The HTML then uses the ?nes= parameter to pass the command line argument:

var argument;
if (QueryString.hasOwnProperty("nes")) {
  argument = QueryString.nes;
} else {
  argument = "smb3.nes";
}

var Module;
Module = {
  preRun: [],
  postRun: [],
  arguments: [argument],
  canvas: (function() {
    var canvas = document.getElementById('canvas');
    canvas.addEventListener("webglcontextlost", function(e) { alert('WebGL context lost. You will need to reload the page.'); e.preventDefault(); }, false);
    return canvas;
  })(),
  totalDependencies: 0
};

Inside the Nim source code there are some interesting changes too. I quickly wrapped these functions as there is no emscripten wrapper for Nim yet:

when defined(emscripten):
  proc emscripten_set_main_loop(fun: proc() {.cdecl.}, fps,
    simulate_infinite_loop: cint) {.header: "<emscripten.h>".}

  proc emscripten_cancel_main_loop() {.header: "<emscripten.h>".}

Emscripten requires a slightly different execution style. Instead of actually looping, we define the main loop like this:

when defined(emscripten):
  emscripten_set_main_loop(loop, 0, 1)
else:
  while runGame:
    loop()

That’s the main idea and with this we get a pretty playable web version of NimES. I’m still getting 60fps in it, but just barely on my machine. Chrome seems to do a bit better than Firefox.

Android port

Obviously the next step is to port NimES to Android as well. But since the original emulator is more accurate and nice than performant, we shouldn’t expect runnable speed. Think of this more as a proof of concept:

We need a fresh clone of the SDL2 repository for this as well as the Android SDK (12 or later) and NDK (7 or later) installed. SDL2 has building instructions for Android as well:

$ hg clone https://hg.libsdl.org/SDL
$ cd SDL/build-scripts
$ ./androidbuild.sh org.nimes /dev/null
$ ls ../build/org.nimes
gen/  src/                 build.properties    local.properties
jni/  AndroidManifest.xml  build.xml           proguard-project.txt
res/  ant.properties       default.properties  project.properties

That’s our Android build directory now. I put this into the repository as well, under android/. Now we can add some ROM to the assets/ directory and tell Nim to put the resulting C files into the correct directory and not to build them into binaries at all:

@if android:
  cpu = "i386"
  nimcache = "./android/jni/src"
  compileOnly
  noMain
@end

You may have noticed that I also defined noMain. Instead we define our own main function, as SDL is a bit weird with mains. Thanks to yglukhov for this little trick:

when defined(android):
  {.emit: """
  #include <SDL_main.h>

  extern int cmdCount;
  extern char** cmdLine;
  extern char** gEnv;

  N_CDECL(void, NimMain)(void);

  int main(int argc, char** args) {
      cmdLine = args;
      cmdCount = argc;
      gEnv = NULL;
      NimMain();
      return nim_program_result;
  }

  """.}

Another trick is how to access the assets we embed into our APK. Luckily SDL2 provides functions for that, which we can use as replacements for the regular file operations:

from sdl2 import rwFromFile, read, freeRW

proc newCartridge*(path: string): Cartridge =
  var file = rwFromFile(path.cstring, "r")
  defer: freeRW file

  var header: iNESHeader
  # Read directly into the header object
  if read(file, addr header, 1, sizeof header) != sizeof header:
    raise newException(ValueError, "header can't be read")
  ...

Finally we can build the project:

$ nim -d:release -d:android c src/nimes
$ cd android
$ ndk-build
$ ant debug

And the end result is a nice nimes.apk. Of course it only shows some low FPS video for now and doesn’t even have any controls, but it’s a start.

Conclusion

In the end I’m quite happy with the result: A truly portable emulator written in my favorite language. It compiles to C, C++ as well as JavaScript and runs on any Desktop platform as well as JavaScript and Android. The process for this was much easier than expected, mostly thanks to Nim and SDL2. I see a bright future for Nim as a practical language.

If you have any comments, suggestions or questions, feel free to ask them on Hacker News or Reddit.

NimES: a NES emulator in Nim

2015-04-30T00:00:00-04:00

Since today Nim 0.11 has been released, I guess it’s a good time to release this as well: NimES is a NES emulator written in Nim. More info in the GitHub repository and the JS demo.

Discussion on Hacker News and Reddit.

Make a Lisp in Nim

2015-03-04T00:00:00-05:00

I spent the last weekend working through the amazing guide for Make a Lisp, writing a Lisp interpreter in Nim. The final result just made it into the repository.

Running the Nim version is pretty simple. You need the Nim compiler from the devel branch and nre which can be installed through nimble. After installing those you can build and run the MAL interpreter:

$ cd nim
$ make
  # OR
$ nimble build
$ ./stepA_mal
Mal [nim]
user> 12
12
user> (+ 2 3)
5

Running the tests:

$ cd ..
$ make "test^nim^step1" # A single test
$ make "test^nim" # All tests
$ make "perf^nim" # All benchmarks

There is a nice presentation in MAL about MAL you can read:

$ cd nim
$ ./stepA_mal ../examples/clojurewest2014.mal

And you can run MAL implented in MAL itself using the Nim interpreter:

$ ./stepA_mal ../mal/stepA_mal.mal
Mal [nim-mal]
mal-user>

The benchmark results from Joel Martin, the author of MAL, don’t look bad for Nim. Edit: Note that these are just rough measurements to see that the Nim implementation is doing fine. Don’t judge the other languages for their numbers, which may not have ideal implementations performance-wise.

In the short benchmarks Nim is the fastest, in the long benchmark only the JVM can beat it. I didn’t really try to optimize and oriented mostly on the Python implementation, which is quite a lot slower as you can see. So it’s pretty nice to see idiomatic Nim performing well:

             perf1  perf2       perf3
            macros   math      macros
                ms     ms   iters/sec
java             6     24       17969
scala           47     87       15963
nim              1      1       11121
ocaml            1      3        7063
cs              10     11        5414
vb              12     13        4523
rust             2      5        4084
c                1      4        3649
go               1      6        3048
racket           3     10        2461
coffee           5     11        2326
js               6     14        1726
ruby             4     15        1255
haskell          4     14        1163
clojure         11     23        1174
forth            9     29         563
php             13     51         331
python          14     51         304
bash          1673   9000         276
perl            18     69         215
ps              41    332          48
R              116    490          28
miniMAL        779   3448           4
matlab        1688   5844           2
make          3427  28453           0

I also didn’t really aim for a low amount of code but rather good readability, but it may still be interesting to look at. I guess Nim’s size being so close to Python shows how much I oriented on the Python implementation:

                Lines   Words   Chars
mal               280     913    7075
clojure           369    1164   10099
ruby              481    1513   13247
coffee            532    2020   15653
racket            550    1937   17229
python            670    1997   19632
nim               715    2453   20263
r                 812    2283   21644
lua               914    2640   23102
ocaml             597    3704   24371
scala             871    3060   24885
js                920    3059   26437
php               940    3011   27332
matlab            959    2485   28322
perl             1081    3511   29091
miniMAL           857    3104   29983
haskell           985    4452   30115
bash             1347    3586   31717
go               1297    4744   36321
ps               1409    6117   38651
forth            1609    6826   44715
cs               1249    4136   45039
java             1591    4966   53223
rust             1897    5702   56516
vb               1556    5175   58099
make             1593    6055   62310
c                2304    6957   73047

Thanks to Joel Martin for this guide. This was my first time getting close to Lisp and I recommend others to work through the MAL guide as well. But be aware, once you get through the first steps, it becomes addictive.

Comments on Reddit and Hacker News.

How I Start: Nim

2015-02-09T00:00:00-05:00

My most recent post goes in another direction (no, it’s still about Nim): How I start a new Nim project. Check out the article on howistart.org. Comments on Hacker News and Reddit as always.

Conclusion on Nim

2015-01-26T00:00:00-05:00

In my last two posts, “What is special about Nim?” and “What makes Nim practical”, I forgot the important conclusion - why I personally have decided for Nim in favor of Rust, C++, Python and Haskell:

Nim is not the fastest language, it’s not the easiest language to write in and it surely has some flaws that should be fixed. Nim has no single “killer feature” like go’s goroutines or Rust’s memory management. But Nim doesn’t need a killer feature. Instead it strikes a reasonable balance that makes it the most efficient language for me:

I can produce reasonably efficient code (faster than Python and Haskell)
that is reasonably readable (more so than Rust, C++ and Haskell)
in a reasonable amount of time (less than Rust, C++ and Haskell)

What makes Nim practical?

2015-01-23T00:00:00-05:00

In my last post I showed what makes the Nim programming language special. Today, let’s consider Nim from another angle: What makes Nim a practical programming language?

Binary Distribution

Programs written in interpreted languages like Python are difficult to distribute. Either you require Python (in a specific version even) to be installed already, or you ship it with your program. This even causes some to reconsider Python as a teaching language.

How does Nim work around this problem? For starters your program gets statically linked against the Nim runtime. That means you end up with a single binary that depends solely on the standard C library, which we can take for granted on any operating system we’re interested in.

Let’s write a small program and give this a try:

echo "Hello World"

If you want to follow along, get the Nim compiler. Save this code as hello.nim. Let’s compile it now so that we can distribute the hello binary:

$ nim -d:release c hello
CC: hello
CC: stdlib_system
[Linking]
$ ./hello
Hello World
$ ls -lha hello
-rwxr-xr-x 1 def def 57K Jan 22 09:37 hello*
$ ldd hello
    linux-vdso.so.1 (0x00007fffd5973000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f0f92c6b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0f928c3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0f92e6f000)

Now our little program starts growing and we get interested in using a few Nim libraries. Do we have to add compiled versions of these libraries to our distributions now? Do not fret! Nim libraries are statically compiled into our binary as well. Let’s get a library using Nim’s package manager, nimble:

$ nimble install strfmt
Installing strfmt-0.5.4
strfmt installed successfully.

And use the library in our, admittedly not very useful, program:

import strfmt

echo "Hello {} number {:04.1f}".fmt("World", 6.0)

If we want to compile with Clang instead of GCC as the backend, that’s easy as well and Clang is usually much faster to compile and yields a smaller binary:

$ nim --cc:clang -d:release c hello
$ ./hello
Hello World number 06.0
$ ls -lha hello
-rwxr-xr-x 1 def def 54K Jan 22 10:06 hello*

The only dynamic libraries we have to worry about are the C libraries we use, but that’s a story for another time.

Source Distribution

We can’t compile our program for all possible combinations of operating systems and CPU architectures of course. But we can be prepared for them and create a source distribution that can be compiled on the target systems with just a C compiler:

$ tree
.
├── hello.ini
├── hello.nim
└── lib
    └── nimbase.h
$ cat hello.ini
[Project]
Platforms: """
  windows: i386;amd64
  linux: i386;amd64;powerpc64;arm;sparc;mips;powerpc
  macosx: i386;amd64;powerpc64
  solaris: i386;amd64;sparc
  freebsd: i386;amd64
  netbsd: i386;amd64
  openbsd: i386;amd64
  haiku: i386;amd64
"""

[Lib]
Files: "lib/nimbase.h"

[Windows]
binPath: "bin"
$ niminst csource hello.ini -d:release

Here’s the resulting build script if you want to try it out on any of these platforms: hello-csources.zip

$ unzip hello-csources.zip
$ ./build.sh
SUCCESS
$ bin/hello
Hello World

niminst can also be used to build full installers for your program. It’s what the Nim compiler itself uses.

Debug and Release Builds

Debug and release builds behave quite differently in Nim by default. Here’s an overview:

Option	Debug build	Release build
objChecks	On	Off
fieldChecks	On	Off
rangeChecks	On	Off
boundChecks	On	Off
overflowChecks	On	Off
assertions	On	Off
stackTrace	On	Off
lineTrace	On	Off
lineDir	On	Off
deadCodeElim	Off	On

As you develop your program you will be thankful for the range, bound and overflow checks. As you release your program your users will be thankful for the additional speed of disabling those.

If this is not what you want, and you require runtime checks even in release builds, you can enable them project wide or for parts of your code.

For example here we have an expensive max function that takes most of the running time:

proc max[T](x: varargs[T]): T =
  if x.len == 0:
    return

  result = x[0]

  for i in 1 .. x.high:
    if result < x[i]:
      result = x[i]

var s = newSeq[int]()
for i in 1..1_000:
  s.add(i)

echo max(s)

We want our program to never access out of sequence bounds, so we want to compile with nim -d:release --checks:on max. As it’s unreasonable to write this every time, we can create a max.nim.cfg file in our directory instead and enable runtime checks for every compilation of our program:

checks: on

Now we run into the problem that the max function runs too slowly (it doesn’t, but let’s imagine). So we disable runtime checks just for this function:

{.push checks: off.}
proc max[T](x: varargs[T]): T =
  if x.len == 0:
    return

  result = x[0]

  for i in 1 .. x.high:
    if result < x[i]:
      result = x[i]
{.pop.}

Great, now we can control which part of our program has runtime checks and which doesn’t. checks enables and disables all runtime checks at once, but there are more fine-grained controls as well.

Nim instead of Python + C++

High performance languages like C++ may require some boilerplate. A higher level language can be used at compile time to automatically create the boilerplate. In DDNet (which inherited them from Teeworlds) there are many examples for this.

A really simple use case is to get the current git revision at compile time in Python and put it into the config.h with a #define so it can be referred to at runtime in the program. In Nim you need neither Python nor a C preprocessor and can instead do it all directly at compile time in Nim:

const gitHash = staticExec "git rev-parse HEAD"
echo "Revision: ", gitHash

Use as a scripting language

With TinyCC as the backend Nim makes for a nice scripting language. All we need is a nimscript file and the TinyCC compiler:

#!/bin/sh
nim --cc:tcc --verbosity:0 -d:release -r c $*

To automatically update DDNet’s map files on our upcoming HTTP map server I’m using this script (with this crc32 module):

#!/usr/bin/env nimscript

import os, crc32, strutils

const
  baseDir = "/home/teeworlds/servers"
  mapdlDir = "/var/www-maps"

for kind, path in walkDir baseDir/"maps":
  if kind != pcFile:
    continue

  let (dir, name, ext) = splitFile(path)

  if ext != ".map":
    continue

  let
    sum = crc32FromFile(path).int64.toHex(8).toLower
    newName = name & "_" & sum & ext
    newPath = mapdlDir / newName
    tmpPath = newPath & ".tmp"

  if existsFile newPath:
    continue

  copyFile path, tmpPath
  moveFile tmpPath, newPath

Of course you can always compile your program if you need the full speed of Clang or GCC. But for quick tests while developing, or scripts you just need occasionally, this is good enough.

There is a nicer version of this idea now, called nimrun

Debugging Nim

GDB works rather well with Nim. The only problem I’ve had so far is that the variables get unique identifiers in C, but using tab completion you can get along fine:

echo "Hello World"
var x = 10
echo "Value of x: ", x

$ nim --lineDir:on --debuginfo c hello
$ gdb ./hello
(gdb) break hello.nim:3
Breakpoint 1 at 0x41f886: file /home/def/hello.nim, line 3.
(gdb) run
Starting program: /home/def/hello
Hello World
Breakpoint 1, helloInit () at /home/def/hello.nim:3
3       echo "Value of x: ", x
(gdb) print x_89010
$1 = 10
(gdb) print x_89010 = 200
$2 = 200
(gdb) c
Continuing.
Value of x: 200

Wrapping libraries with c2nim

Nice C libraries can automatically be converted into a Nim wrapper with c2nim. Let’s do this for Bellard’s new BPG image format. Since the library is so new, there is no shared library compilation included yet, so I added that myself in my fork.

We need to fix up the C header for c2nim by adding this at the top of libbpg.h:

#ifdef C2NIM
#  dynlib bpglib
#  cdecl

/* Dynamically link to the correct library for our system: */
#  if defined(windows)
#    define bpglib "libbpg.dll"
#  elif defined(macosx)
#    define bpglib "libbpg.dylib"
#  else
#    define bpglib "libbpg.so"
#  endif

/* Remove prefixes in our wrapper, we have modules in Nim: */
#  prefix bpg_decoder_
#  prefix BPG_
#  prefix BPG

/* These are not recognized by c2nim, but that's easy to fix: */
#  mangle uint8_t uint8
#  mangle uint16_t uint16
#  mangle uint32_t uint32
#endif

I made a patch so I can apply the changes again when the libbpg interface changes. Creating the wrapper is now as simple as this:

$ patch -p1 < libbpg.h.patch
patching file libbpg.h
$ c2nim -o:bpg.nim libbpg.h
Hint: operation successful (155 lines compiled; 0 sec total; 516.528KB) [SuccessX]

Look, no hands! The resulting wrapper doesn’t even look bad. Now we can write a simple program to decode a BPG file and save it in the PPM format:

import bpg, os

proc writePPM(img, filename) =
  var imgInfo: ImageInfo
  discard img.getInfo(addr imgInfo)

  let (w,h) = (imgInfo.width.int, imgInfo.height.int)
  var rgbLine = newSeq[uint8](w * 3)

  var f = open(filename, fmWrite)
  f.writeln "P6\n", w, " ", h, "\n255"

  discard img.start(OUTPUT_FORMAT_RGB24)
  for y in 1..h:
    discard img.getLine(addr rgbLine[0])
    discard f.writeBuffer(addr rgbLine[0], w * 3)

  f.close()

if paramCount() != 1:
  stderr.writeln "Usage: decode img.bpg"
  quit 1

var
  buf = readFile paramStr(1)
  img = bpg.open()

if img.decode(cast[ptr uint8](addr buf[0]), buf.len.cint) < 0:
  stderr.writeln "Could not decode image"
  quit 2

img.writePPM("out.ppm")
img.close()

Note that you can tell from the discard statements where I chose to ignore errors. If you actually want to run this, check out the instructions in the repository.

Final words

That’s Nim from a more practical angle. Hopefully you’ll consider Nim for your next project, there are many libraries available already. Also, the community always needs more helping hands!

To do my part in making Nim more practical, I’m trying to implement a new, working REPL using TinyCC. We may also soon get a working compiler as a service for proper IDE integration, directly from Andreas Rumpf.

For comments use Reddit, Hacker News or get in touch with the Nim community directly. You can reach me personally at dennis@felsing.org.

Thanks to Andreas Rumpf and Dominik Picheta again, for proof-reading this post as well.

What is special about Nim?

2015-01-01T00:00:00-05:00

Russian Translation by frol, Chinese Translation by JiyinYiyong, Japanese Translation by Mutsuha Asada

The Nim programming language is exciting. While the official tutorial is great, it slowly introduces you to the language. Instead I want to quickly show what you can do with Nim that would be more difficult or impossible in other languages.

I discovered Nim in my quest to find the right tools to write a game, HookRace, the successor of my current DDNet game/mod of Teeworlds. Since I’m busy with other projects for now, this blog is now officially about Nim instead, until I find time to continue developing the game.

Easy to get running

Ok, this part is not exciting yet, but I invite you to follow along with the post:

for i in 0..10:
  echo "Hello World"[0..i]

If you want to do so, get the Nim compiler. Save this code as hello.nim, compile it with nim c hello and finally run the binary with ./hello. To immediately compile and run, use nim -r c hello. To use an optimized release build instead of a debug build use nim -d:release c hello. With all of these settings you will see the following output:

H
He
Hel
Hell
Hello
Hello 
Hello W
Hello Wo
Hello Wor
Hello Worl
Hello World

Run regular code at compile time

To implement an efficient CRC32 procedure you need a lookup table. You could compute it at runtime or write it into your code as a magic array. Clearly we don’t want any magic numbers in our code, so we’ll do it at runtime (for now):

import unsigned, strutils

type CRC32* = uint32
const initCRC32* = CRC32(-1)

proc createCRCTable(): array[256, CRC32] =
  for i in 0..255:
    var rem = CRC32(i)
    for j in 0..7:
      if (rem and 1) > 0: rem = (rem shr 1) xor CRC32(0xedb88320)
      else: rem = rem shr 1
    result[i] = rem

# Table created at runtime
var crc32table = createCRCTable()

proc crc32(s): CRC32 =
  result = initCRC32
  for c in s:
    result = (result shr 8) xor crc32table[(result and 0xff) xor ord(c)]
  result = not result

# String conversion proc $, automatically called by echo
proc `$`(c: CRC32): string = int64(c).toHex(8)

echo crc32("The quick brown fox jumps over the lazy dog")

Great, this works and we get 414FA339 as the output. But it would be even better if we could compute the CRC table at compile time. This is as easy as it gets in Nim, instead of the current crc32table creation we use:

# Table created at compile time
const crc32table = createCRCTable()

Yes, that’s right: All we had to do was replace the var with a const. Beautiful, isn’t it? We can write the exact same code and have it run at runtime and compile time. No template metaprogramming necessary.

Extend the language

Templates and macros can be used to get rid of boilerplate, by transforming your code at compile time.

Templates just replace their invocations with their code at compile-time. We can define our own loops like this:

template times(x: expr, y: stmt): stmt =
  for i in 1..x:
    y

10.times:
  echo "Hello World"

So the compiler transforms the times-loop to this regular for-loop:

for i in 1..10:
  echo "Hello World"

If you’re curious about the 10.times: syntax, it’s just a regular call to times with 10 as the first parameter and the following indented block as the second parameter. Instead you could also write times(10):, see Unified Call Syntax.

Or initialize sequences (variable sized arrays) more comfortably:

template newSeqWith(len: int, init: expr): expr =
  var result = newSeq[type(init)](len)
  for i in 0 .. <len:
    result[i] = init
  result

# Create a 2-dimensional sequence of size 20,10
var seq2D = newSeqWith(20, newSeq[bool](10))

import math
randomize()
# Create a sequence of 20 random integers smaller than 10
var seqRand = newSeqWith(20, random(10))
echo seqRand

Macros go a step further and allow you to analyze and manipulate the AST. There are no list comprehensions in Nim, for example, but it’s possible to add them to the language by using a macro. Now instead of this:

var res: seq[int] = @[]
for x in 1..10:
  if x mod 2 == 0:
    res.add(x)
echo res

const n = 20
var result: seq[tuple[a,b,c: int]] = @[]
for x in 1..n:
  for y in x..n:
    for z in y..n:
      if x*x + y*y == z*z:
        result.add((x,y,z))
echo result

You can use the future module and write:

import future
echo lc[x | (x <- 1..10, x mod 2 == 0), int]
const n = 20
echo lc[(x,y,z) | (x <- 1..n, y <- x..n, z <- y..n,
                   x*x + y*y == z*z), tuple[a,b,c: int]]

Add your own optimizations to the compiler

Instead of optimizing your code, wouldn’t you prefer to make the compiler smarter? In Nim you can!

var x: int
for i in 1..1_000_000_000:
  x += 2 * i
echo x

This (pretty useless) code can be sped up by teaching the compiler two optimizations:

template optMul{`*`(a,2)}(a: int): int =
  let x = a
  x + x

template canonMul{`*`(a,b)}(a: int{lit}, b: int): int =
  b * a

In the first term rewriting template we specify that a * 2 can be replaced by a + a. In the second one we specify that ints in a multiplication can be swapped if the first is an integer literal, so that we can potentially apply the first template.

More complicated patterns can also be implemented, for example to optimize boolean logic:

template optLog1{a and a}(a): auto = a
template optLog2{a and (b or (not b))}(a,b): auto = a
template optLog3{a and not a}(a: int): auto = 0

var
  x = 12
  s = x and x
  # Hint: optLog1(x) --> ’x’ [Pattern]

  r = (x and x) and ((s or s) or (not (s or s)))
  # Hint: optLog2(x and x, s or s) --> ’x and x’ [Pattern]
  # Hint: optLog1(x) --> ’x’ [Pattern]

  q = (s and not x) and not (s and not x)
  # Hint: optLog3(s and not x) --> ’0’ [Pattern]

s gets optimized to x directly, r gets optimized to x in 2 successive pattern applications and q ends up as 0 immediately.

If you want to see how term rewriting templates can be used to avoid allocations with bigints, look for the templates starting with opt in the bigints library:

import bigints

var i = 0.initBigInt
while true:
  i += 1
  echo i

Bind to your favorite C functions and libraries

Since Nim compiles down to C, foreign function interfaces are fun.

You can easily use your favorite functions from the C standard library:

proc printf(formatstr: cstring)
  {.header: "<stdio.h>", varargs.}
printf("%s %d\n", "foo", 5)

Or use your own code written in C:

void hi(char* name) {
  printf("awesome %s\n", name);
}

{.compile: "hi.c".}
proc hi*(name: cstring) {.importc.}
hi "from Nim"

Or any library you please with the help of c2nim:

proc set_default_dpi*(dpi: cdouble) {.cdecl,
  importc: "rsvg_set_default_dpi",
  dynlib: "librsvg-2.so".}

Control the garbage collector

To achieve soft realtime, you can tell the garbage collector when and for how long it is allowed to run. The main game logic can be implemented like this in Nim, to prevent the garbage collector from causing stutters:

gcDisable()
while true:
  gameLogic()
  renderFrame()
  gcStep(us = leftTime)
  sleep(restTime)

Type safe sets and arrays of enums

Often you want a mathematical set over values you defined yourself. Here’s how you do this in type-safe way:

type FakeTune = enum
  freeze, solo, noJump, noColl, noHook, jetpack

var x: set[FakeTune]

x.incl freeze
x.incl solo
x.excl solo

echo x + {noColl, noHook}

if freeze in x:
  echo "Here be freeze"

var y = {solo, noHook}
y.incl 0 # Error: type mismatch

You can’t accidentally add a value of another type. Internally the set works as an efficient bitvector.

The same is possible with arrays, indexing them with your enum:

var a: array[FakeTune, int]
a[freeze] = 100
echo a[freeze]

Unified Call Syntax

This is just syntactic sugar, but it’s definitely nice to have. In Python I always forget whether len and append are functions or methods. In Nim I don’t have to remember, because there is no difference. Nim uses Unified Call Syntax, which has now also been proposed for C++ by Herb Sutter and Bjarne Stroustrup.

var xs = @[1,2,3]

# Procedure call syntax
add(xs, 4_000_000)
echo len(xs)

# Method call syntax
xs.add(0b0101_0000_0000)
echo xs.len()

# Command invocation syntax
xs.add 0x06_FF_FF_FF
echo xs.len

Good performance

It’s easy to write fast code in Nim, as can be seen in the Longest Path Finding Benchmark, in which Nim is competing with some cute code.

I made some measurements on my machine when the benchmark was first published (Linux x86-64, Intel Core2Quad Q9300 @2.5GHz, state of 2014-12-20):

Lang	Time [ms]	Memory [KB]	Compile Time [ms]	Compressed Code [B]
Nim	1400	1460	893	486
C++	1478	2717	774	728
D	1518	2388	1614	669
Rust	1623	2632	6735	934
Java	1874	24428	812	778
OCaml	2384	4496	125	782
Go	3116	1664	596	618
Haskell	3329	5268	3002	1091
LuaJit	3857	2368	-	519
Lisp	8219	15876	1043	1007
Racket	8503	130284	24793	741

Compressed code size with gzip -9 < nim.nim | wc -c. Removed unused functions in Haskell. Compile times are clean compiles, if you have a nimcache with the standard library precompiled it’s only 323 ms for Nim.

Another small benchmark I did, calculating which numbers of the first 100 million are prime, in Python, Nim and C:

Python (runtime: 35.1 s)

def eratosthenes(n):
  sieve = [1] * 2 + [0] * (n - 1)
  for i in range(int(n**0.5)):
    if not sieve[i]:
      for j in range(i*i, n+1, i):
        sieve[j] = 1
  return sieve

eratosthenes(100000000)

Nim (runtime: 2.6 s)

import math

proc eratosthenes(n): auto =
  result = newSeq[int8](n+1)
  result[0] = 1; result[1] = 1

  for i in 0 .. int sqrt(float n):
    if result[i] == 0:
      for j in countup(i*i, n, i):
        result[j] = 1

discard eratosthenes(100_000_000)

C (runtime: 2.6 s)

#include <stdlib.h>
#include <math.h>
char* eratosthenes(int n)
{
  char* sieve = calloc(n+1,sizeof(char));
  sieve[0] = 1; sieve[1] = 1;
  int m = (int) sqrt((double) n);

  for(int i = 0; i <= m; i++) {
    if(!sieve[i]) {
      for (int j = i*i; j <= n; j += i)
        sieve[j] = 1;
    }
  }
  return sieve;
}

int main() {
  eratosthenes(100000000);
}

Compile to JavaScript

You can compile Nim to JavaScript, instead of C. This allows you to write the client as well as the server directly in Nim. Let’s make a small visitor counter on the server that gets displayed in the browser. This is our client.nim:

import htmlgen, dom

type Data = object
  visitors {.importc.}: int
  uniques {.importc.}: int
  ip {.importc.}: cstring

proc printInfo(data: Data) {.exportc.} =
  var infoDiv = document.getElementById("info")
  infoDiv.innerHTML = p("You're visitor number ", $data.visitors,
    ", unique visitor number ", $data.uniques,
    " today. Your IP is ", $data.ip, ".")

We define a Data type that we use to pass data from the server to client. The printInfo procedure will be called with this data and display it. Compile this with nim js client. The result javascript file ends up in nimcache/client.js.

For the server we need to get the Nimble package manager and run nimble install jester. Now we can use the Jester web framework and write our server.nim:

import jester, asyncdispatch, json, strutils, times, sets, htmlgen, strtabs, httpcore

var
  visitors = 0
  uniques = initSet[string]()
  time: TimeInfo

routes:
  get "/":
    resp body(
      `div`(id="info"),
      script(src="/client.js", `type`="text/javascript"),
      script(src="/visitors", `type`="text/javascript"))

  get "/client.js":
    const result = staticExec "nim -d:release js client"
    const clientJS = staticRead "nimcache/client.js"
    resp clientJS

  get "/visitors":
    let newTime = getTime().getLocalTime
    if newTime.monthDay != time.monthDay:
      visitors = 0
      init uniques
      time = newTime

    inc visitors
    let ip =
      if request.headers.hasKey "X-Forwarded-For":
        request.headers["X-Forwarded-For", 0]
      else:
        request.ip
    uniques.incl ip

    let json = %{"visitors": %visitors,
                 "uniques": %uniques.len,
                 "ip": %ip}
    resp "printInfo($#)".format(json)

runForever()

The server delivers the main website. It also delivers the client.js, by compiling and reading the client.nim at compile time. The logic is in the /visitors handling. Compile and run with nim -r c server and open http://localhost:5000/ to see the code in effect.

You can see our code in action on the Jester-generated site or inline here:

You don't have JavaScript enabled or something went wrong.

Final words

I hope I could pique your interest in in the Nim programming language.

Note that the language is not completely stable yet. Especially with the more obscure features you may run into bugs. But on the bright side, Nim 1.0 is supposed to be released within the next 3 months! So now is the perfect time to get started with learning Nim.

Bonus: Since Nim compiles to C and only depends on the C standard library you can deploy it nearly everywhere, including x86-64, ARM and Intel Xeon Phi accelerator cards.

For comments use Reddit, Hacker News or ask the Nim community directly on IRC (#nim on freenode). You can reach me personally at dennis@felsing.org.

Thanks to Andreas Rumpf and Dominik Picheta for proof-reading this post.

Nim Adventures

2014-07-13T00:00:00-04:00

To learn some Nim I held a talk about it at the GPN14 (in German, Slides).

Afterwards I started solving Rosetta Code tasks in Nim to get a better feel for the language and standard library. That also made me discover some rough edges in the language, but luckily the community, albeit small, is active and competent. All the small code pieces I wrote are now on Github too.

What I noticed is that most problems are as easy to solve as in Python, but much more performant. I’m now more confident that this language is the right choice for writing HookRace in.

Some highlights:

Insertion Sort

Once in Python:

def insertion_sort(l):
    for i in xrange(1, len(l)):
        j = i-1 
        key = l[i]
        while (l[j] > key) and (j >= 0):
           l[j+1] = l[j]
           j -= 1
        l[j+1] = key

and in Nim:

proc insertSort[T](a: var openarray[T]) =
  for i in 1 .. <a.len:
    let value = a[i]
    var j = i
    while j > 0 and value < a[j-1]:
      a[j] = a[j-1]
      dec j
    a[j] = value

Rank languages by popularity

The standard library is extremely useful, providing an HTTP client, JSON parser, regular expressions, string utils and algorithms, which I use to create a ranking of the popularity of languages on Rosetta Code:

import httpclient, json, re, strutils, algorithm, future

const
  langSite = "http://www.rosettacode.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Programming_Languages&cmlimit=500&format=json"
  catSize = "http://www.rosettacode.org/w/index.php?title=Special:Categories&limit=5000"
let regex = re"title=""Category:(.*?)"">.+?</a>.*\((.*) members\)"

var langs = newSeq[string]()
for l in parseJson(getContent(langSite))["query"]["categorymembers"]:
  langs.add(l["title"].str.split("Category:")[1])

var ranks = newSeq[tuple[lang: string, count: int]]()
for line in getContent(catSize).findAll(regex):
  let lang = line.replacef(regex, "$1")
  if lang in langs:
    let count = parseInt(line.replacef(regex, "$2").strip())
    ranks.add((lang, count))

ranks.sort((x, y) => cmp[int](y.count, x.count))
for i, l in ranks:
  echo align($(i+1), 3), align($l.count, 5), " - ", l.lang

Calling a C function

proc printf(formatstr: cstring) {.header: "<stdio.h>", varargs.}

var x = "foo"
printf("Hello %d %s!\n", 12, x)

Wrapping a shared C library

when defined(windows):
  const imglib = "imglib.dll"
elif defined(macosx):
  const imglib = "imglib.dylib"
else:
  const imglib = "imglib.so"

proc openimage(s: cstring): cint {.importc, dynlib: imglib.}

echo openimage("foo")

Quine

A quine is a program that prints its own source code. This Nim program does that once at compiletime and once at runtime:

const x = "const x = |const y = x[0..9]&34.chr&x&34.chr&10.chr&x[11..100]&10.chr&x[102..115]&10.chr&x[117 .. -1]|static: echo y|echo y"
const y = x[0..9]&34.chr&x&34.chr&10.chr&x[11..100]&10.chr&x[102..115]&10.chr&x[117 .. -1]
static: echo y
echo y

What is HookRace?

2014-07-07T00:00:00-04:00

I have been running DDraceNetwork for 1 year, which started out as a little server of a Teeworlds modification called DDRace. It has turned into the biggest project within Teeworlds, offering servers in 6 countries with thousands of players and dozens of mappers. We also offer a custom server and client (Windows, Linux, OS X, Android).

Now it’s time to start something new. I’m working on a new game called HookRace, which will be the successor to DDraceNetwork. It will be its own game, not based on Teeworlds.

Some of the plans I have so far for HookRace:

Vector Graphics (SVG) ingame for a sharp image on any screen
Whole physics is executed at the client as well as the server, for perfect prediction
No lags with high player numbers
Accounts to prevent faking

As the programming language for HookRace, I chose Nim:

# Game of Life in terminal
import os, strutils, math

randomize()

var w, h: int
if paramCount() >= 2:
  w = parseInt paramStr 1
  h = parseInt paramStr 2
if w <= 0: w = 30
if h <= 0: h = 30

# Iterate over fields in the universe
iterator fields(a = (0,0), b = (h-1,w-1)) =
  for y in a[0]..b[0]:
    for x in a[1]..b[1]:
      yield (y,x)

# Create a sequence with an initializer
proc newSeqWith[T](len: int, init: T): seq[T] =
  result = newSeq[T] len
  for i in 0 .. <len:
    result[i] = init

# Initialize
var univ, univNew = newSeqWith(h, newSeq[bool] w)

for y,x in fields():
  if random(10) < 1: univ[y][x] = true

while true:
  # Show
  stdout.write "\e[H"
  for y,x in fields():
    stdout.write if univ[y][x]: "\e[07m  \e[m" else: "  "
    if x == 0: stdout.write "\e[E"
  stdout.flushFile

  # Evolve
  for y,x in fields():
    var n = 0
    for y1,x1 in fields((y-1,x-1), (y+1,x+1)):
      if univ[(y1+h) mod h][(x1 + w) mod w]:
        inc n

    if univ[y][x]: dec n
    univNew[y][x] = n == 3 or (n == 2 and univ[y][x])
  swap univ, univNew

  sleep 200

Nim is a rather new language, and it took me some time to choose it. What I love about the language:

Python-like syntax
Statically typed, compiled
High performance (same ballpark as C/C++)
Garbage Collector that can be controlled for soft real-time
Produces executables without dependencies (compiles to C)
Easily interfaces C libraries (compiles to C)
Clean and powerful metaprogramming

On this blog I will report about my progress in the development of HookRace and my current Nim adventures.

Programming Posts · HookRace Blog

Workload Capture & Replay

Speeding up Materialize CI

How to Use the Materialize Emulator

The CI Flake

Testing Materialize

Haskell: Game Programming with GIF Streams

Printing the Playing Field

Snake Behavior

Food Behavior

End of the Game

Bonus

God writes Haskell

Database Testing at Yugabyte

macOS Setup after 15 Years of Linux

Installing Software

Command line software

Window management

Terminal

Keyboard layout

Deadkeys

Mouse

Task bar

Firefox & Web

Music

Translations

Standby

Useful shortcuts

Conclusion

Quest for the Lost Home Server

Chinese Traffic to time.gif

Quantifying the Traffic

Keeping Up with the Traffic

Final Words

Linux Desktop Setup

Motivation

Operating Systems

Window Manager

Terminal Persistency

Shell & Programming

System Monitoring

Mails & Synchronization

News & Communication

Calendar

Dictionary

Writing and Reading

Graphical Editing

Web Browsing

Media Players

Cross-Platform Building

Backups

Final Remarks

Theme Switcher in Nginx

Turning Down a Blockchain Job Offer

Introduction

The Good

The Bad

The Conclusion

HANA C++ Development Environment and Processes

Compilation

Coding

Local Verification

Remote Verification

Code Review

Merging

Conclusion

C++ Quiz #1

Solution

Explanation

Future Directions

time.gif

Nim Code Coverage

A Taste of Haskell

Arithmetic

Lists

Higher-order Functions

Lazy Evaluation and Sharing

List Comprehensions

Conclusion

Writing a 2D Platform Game in Nim with SDL2