Nim Posts · HookRace Blog

Nim Code Coverage

2016-11-01T00:00:00-04:00

Creating code coverage reports with Nim is surprisingly easy. You can simply use the good old gcov and lcov tools. Nim can be told to insert its own line information with the --debugger:native command line parameter.

Here’s the small example program we’re looking at:

var x = 0
if x > 1:
  echo "foo"
echo "bar"

Note that if we change the condition to if false: the Nim compiler optimizes the impossible code away and it will not count as uncovered. The same thing can happen with entire uncovered functions when optimizations and dead code elimination are enabled.

The file is saved as x.nim. Here’s the script I use to create the code coverage report:

#!/bin/sh
rm -rf *.info html nimcache
nim --debugger:native --passC:--coverage --passL:--coverage c x
lcov --base-directory . --directory . --zerocounters -q
./x
lcov --base-directory . --directory . -c -o x.info
lcov --remove x.info "lib/*" -o x.info # remove Nim system libs from coverage
genhtml -o html x.info

You can look at the final html report generated.

Writing a 2D Platform Game in Nim with SDL2

2016-06-14T00:00:00-04:00

Japanese Translation

In this article we’re going to write a simple 2D platform game. You can also consider this as a tutorial for game development with SDL2 in Nim.

We will read in user input, display graphics and a tile map, and simulate simple 2D physics with collision detection and handling. Afterwards we will implement simple camera movement and game logic. To display some information we will render texts and develop a caching mechanism for said text rendering.

The final result will be a binary file that requires only SDL2 and can be easily distributed, perfect for games. If you’re on Linux we will also present a simple way to cross-compile Nim programs for Windows.

For the sake of simplicity we’re going to use the familiar graphics from DDNet and Teeworlds, with the end result of this tutorial looking like this:

We’re going to follow along with the development throughout this article with illustrative images and videos, but the best way to learn is if you follow along yourself by implementing the steps from this article. The code is purposefully kept simple and easy to extend so that you can play around with it and try out all kinds of changes to get an intuitive understanding. At the end of every section there is a link to its full source code.

The iterations of the code of this article and the final result are available in a repository on GitHub. The resulting binaries can be downloaded here: Win64, Win32, Linux x86_64, Linux x86

Preliminaries

For this post we require:

The Nim programming language and its package manager Nimble
SDL 2, SDL_image 2, SDL_ttf 2 (all for developers)
Nim SDL2 wrapper and strfmt, which can be installed using Nimble

On a unixoid system like Linux or Mac OS X the installation looks something like this:

# Debian / Ubuntu
$ sudo apt-get install git libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev
# Arch Linux
$ pacman -S git sdl2 sdl2_image sdl2_ttf
# Homebrew on OS X
$ brew install git sdl2 sdl2_image sdl2_ttf
# FreeBSD
$ pkg install git sdl2 sdl2_image sdl2_ttf

$ wget http://nim-lang.org/download/nim-0.14.2.tar.xz
$ tar xvf nim-0.14.2.tar.xz
$ cd nim-0.14.2
$ make -j4
$ echo 'export PATH=$HOME/nim-0.14.2/bin:$PATH' >> ~/.profile
$ source ~/.profile

$ git clone https://github.com/nim-lang/nimble.git
$ cd nimble
$ nim -d:release c -r src/nimble install
$ echo 'export PATH=$HOME/.nimble/bin:$PATH' >> ~/.profile
$ source ~/.profile

$ nimble install sdl2 strfmt

Note that you also need a C compiler on your system, preferably GCC or Clang.

Instead of compiling Nim and Nimble from source code you could also use your package manager to install Nim and Nimble if they’re available in a recent version.

For setting up SDL2 on other platforms more extensive guides exist, as do for Nim and nimble.

1. First Running Program

If you have seen SDL2 programs written in C or C++ before, you will notice that what we’re doing in Nim is very similar. Actually Nim’s SDL2 wrapper is just a thin layer wrapping the original SDL2 interface from C. This has the advantage that what you learn from any SDL2 tutorial is applicable, but the disadvantage is that you end up with a bit more boilerplate than with a more high-level Nim library.

We start with exactly this boilerplate to initialize our window:

import sdl2

type SDLException = object of Exception

template sdlFailIf(cond: typed, reason: string) =
  if cond: raise SDLException.newException(
    reason & ", SDL error: " & $getError())

proc main =
  sdlFailIf(not sdl2.init(INIT_VIDEO or INIT_TIMER or INIT_EVENTS)):
    "SDL2 initialization failed"

  # defer blocks get called at the end of the procedure, even if an
  # exception has been thrown
  defer: sdl2.quit()

  sdlFailIf(not setHint("SDL_RENDER_SCALE_QUALITY", "2")):
    "Linear texture filtering could not be enabled"

  let window = createWindow(title = "Our own 2D platformer",
    x = SDL_WINDOWPOS_CENTERED, y = SDL_WINDOWPOS_CENTERED,
    w = 1280, h = 720, flags = SDL_WINDOW_SHOWN)
  sdlFailIf window.isNil: "Window could not be created"
  defer: window.destroy()

  let renderer = window.createRenderer(index = -1,
    flags = Renderer_Accelerated or Renderer_PresentVsync)
  sdlFailIf renderer.isNil: "Renderer could not be created"
  defer: renderer.destroy()

  # Set the default color to use for drawing
  renderer.setDrawColor(r = 110, g = 132, b = 174)

  # Game loop, draws each frame
  while true:
    # Draw over all drawings of the last frame with the default
    # color
    renderer.clear()
    # Show the result on screen
    renderer.present()

main()

We introduced an sdlFailIf template that checks a condition and if the condition is true, raises an SDLException with additional error information from SDL. In the main proc we initialize SDL2, and create a regular window and an accelerated 2D renderer. Error handling is done with the sdlFailIf proc that we introduced.

For now the game loop just clears the window and draws it every frame. If you have VSync enabled and your screen is set to 60 Hz the loop will be executed 60 times per second.

We can compile and run in the same step by executing nim -r c platformer, assuming you called the file platformer.nim. To compile with optimizations use nim -d:release -r c platformer. The result is a simple one-colored window:

We can exit our small program by pressing Ctrl-C in the terminal window. Unfortunately we can’t exit it in the game window itself yet, so let’s fix that.

Full code for section 1

2. User Input

First let’s add an Input type to store all the inputs we want to support, and store an array of Inputs in our game state object:

type
  Input {.pure.} = enum none, left, right, jump, restart, quit

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr

By choosing a ref type for the Game state type we have an easy way to prevent accidentally creating copies of it. By default only the garbage collected pointer to our Game object is passed around. The inputs field is an array mapping from Input to bool, signifying which input is currently pressed (true) or not (false). Creating a new game state object is trivial, we just create a new heap object for now and assign the SDL2 renderer that we will need later:

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

We don’t need to initialize the inputs field in any way as everything is initialized to binary null by default, which is exactly what we want: Every input is set to off in the start. If we didn’t initialize the renderer field it would be a null pointer and we would get into trouble if we accidentally dereference it.

The next thing we need is a procedure that maps keyboard scan codes to our recognized inputs:

proc toInput(key: Scancode): Input =
  case key
  of SDL_SCANCODE_A: Input.left
  of SDL_SCANCODE_D: Input.right
  of SDL_SCANCODE_SPACE: Input.jump
  of SDL_SCANCODE_R: Input.restart
  of SDL_SCANCODE_Q: Input.quit
  else: Input.none

Note that toInput returns Input.none for all undefined cases. We will use this behaviour to ignore unused keyboard inputs without the need for a branch in our code. You could easily recognize multiple scan codes to map to a single input.

We modify the game loop to react to keyboard inputs by calling our new handleInput proc. We also split out the rendering itself to keep the separation of concerns clear in the main proc:

proc handleInput(game: Game) =
  var event = defaultEvent
  while pollEvent(event):
    case event.kind
    of QuitEvent:
      game.inputs[Input.quit] = true
    of KeyDown:
      game.inputs[event.key.keysym.scancode.toInput] = true
    of KeyUp:
      game.inputs[event.key.keysym.scancode.toInput] = false
    else:
      discard

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Show the result on screen
  game.renderer.present()

var game = newGame(renderer)

# Game loop, draws each frame
while not game.inputs[Input.quit]:
  game.handleInput()
  game.render()

Now we can either press q or use the close button on the window to close the game. Other kinds of user input are just stored in the inputs array for now and we will be able to use it later.

Note that the inputs array is intentionally a simple array, so that access to it is as performant as can be. If we used a hash table or some other data structure such a guarantee would not be that easy to make. Simplicity can often be beneficial and make it easier to understand what is going on in the system you’re developing.

Full code for section 2

3. Displaying Graphics

If we want to do something of interest with those user inputs we need to start displaying something other than a blue sky.

We extend our game state object to also store the player texture as well as the current position and velocity, for which we use the basic2d module:

import basic2d

type
  Player = ref object
    texture: TexturePtr
    pos: Point2d
    vel: Vector2d

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    player: Player
    camera: Vector2d

We’re going to use Teeworlds’ default tee graphic for our player:

You can save this file as player.png to follow along. If you feel funny you can also select one of hundreds of skins from the DDNet Skin Database, for example:

Don’t forget to call your player graphic player.png if you want to use an alternative one.

First we have to load the player graphic, whichever one you decided to use, in our definition of newGame and initialize the Game and Player data structures:

import sdl2/image

proc restartPlayer(player: Player) =
  player.pos = point2d(170, 500)
  player.vel = vector2d(0, 0)

proc newPlayer(texture: TexturePtr): Player =
  new result
  result.texture = texture
  result.restartPlayer()

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer
  result.player = newPlayer(renderer.loadTexture("player.png"))

restartPlayer is used to reset the player to its start position. The loadTexture procedure loads the PNG image into memory as an SDL2 texture that we can store in the Game object.

We also shouldn’t forget to initialize the SDL2 image module in our main proc, similarly to the SDL2 initialization:

const imgFlags: cint = IMG_INIT_PNG
sdlFailIf(image.init(imgFlags) != imgFlags):
  "SDL2 Image initialization failed"
defer: image.quit()

We only need support for PNG files, otherwise we could also add JPEG files with const imgFlags: cint = IMG_INIT_PNG or IMG_INIT_JPG.

Next our task is to put this together nicely. Of course the intention of the flexible player images is that parts of the body can move independently, but for the sake of simplicity we will put them into fixed positions. A simple addition would be to make the feet move in a rotating motion depending on the horizontal position of the player. Another addition would be to make the eyes follow the mouse cursor.

proc renderTee(renderer: RendererPtr, texture: TexturePtr,
               pos: Point2d) =
  let
    x = pos.x.cint
    y = pos.y.cint

  var bodyParts: array[8, tuple[source, dest: Rect, flip: cint]] = [
    (rect(192,  64, 64, 32), rect(x-60,    y, 96, 48),
     SDL_FLIP_NONE),      # back feet shadow
    (rect( 96,   0, 96, 96), rect(x-48, y-48, 96, 96),
     SDL_FLIP_NONE),      # body shadow
    (rect(192,  64, 64, 32), rect(x-36,    y, 96, 48),
     SDL_FLIP_NONE),      # front feet shadow
    (rect(192,  32, 64, 32), rect(x-60,    y, 96, 48),
     SDL_FLIP_NONE),      # back feet
    (rect(  0,   0, 96, 96), rect(x-48, y-48, 96, 96),
     SDL_FLIP_NONE),      # body
    (rect(192,  32, 64, 32), rect(x-36,    y, 96, 48),
     SDL_FLIP_NONE),      # front feet
    (rect( 64,  96, 32, 32), rect(x-18, y-21, 36, 36),
     SDL_FLIP_NONE),      # left eye
    (rect( 64,  96, 32, 32), rect( x-6, y-21, 36, 36),
     SDL_FLIP_HORIZONTAL) # right eye
  ]

  for part in bodyParts.mitems:
    renderer.copyEx(texture, part.source, part.dest, angle = 0.0,
                    center = nil, flip = part.flip)

The exact numbers are not so important, they are just how the player is meant to be put together. With renderTee we define which body part is drawn at which position and in which order. Finally each of these body parts is drawn with the SDL2 renderer using copyEx.

Now drawing the tee in our game loop is simply a call to renderTee away:

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  # Show the result on screen
  game.renderer.present()

Finally we have some visual progress again, look at the player floating in the sky:

Full code for section 3

4. Tile Map

Now that we have a working rendering system for the player, we need a map to play in. This requires us to store a texture as well as a list of tiles:

type
  Map = ref object
    texture: TexturePtr
    width, height: int
    tiles: seq[uint8]

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    player: Player
    map: Map
    camera: Vector2d

Each tile is defined to be a uint8, which means a value between 0 and 255 inclusively. Conveniently the tileset graphic for from Teeworlds have 16 × 16 = 256 tiles. We will use the grass tileset:

Download and save this image as grass.png.

To initialize the Map data structure we shall write a little parser that parses maps of this format:

0  0  0 78  0  0  0  0  0  0  0  0  0  0
5  0  0 78  0  0  0  0  0  0  0  0  0  0
21  0  0 78  0  0  0  0  0  0  0  0  0  0
21  0  0 78  0  0  0  0  0  0  0  0  4  5
21  0  0 78  0  0  0  0  0  0  0  0 20 21
21  0  0 78  0  0  0  0  0  0  0  0 20 21
21  0  0 78  0  0  4  5  0  0  0  0 20 21
21  0  0 78  0  0 20 21  0  0  0  0 20 21
38  0  0 78  0  0 22 38  0  0  0  0 22 38
49 16 16 16 16 16 48 49 16 16 16 16 48 49
52 52 52 52 52 52 52 52 52 52 52 52 52 52

Our goal in this section is for this map with the grass tileset to result in this rendering:

Each number denotes the tile that is chosen from the grass tileset. We will use this default.map for the rest of this article. Our parser is implemented in newMap and looks like this:

import strutils

proc newMap(texture: TexturePtr, file: string): Map =
  new result
  result.texture = texture
  result.tiles = @[]

  for line in file.lines:
    var width = 0
    for word in line.split(' '):
      if word == "": continue
      let value = parseUInt(word)
      if value > uint(uint8.high):
        raise ValueError.newException(
          "Invalid value " & word & " in map " & file)
      result.tiles.add value.uint8
      inc width

    if result.width > 0 and result.width != width:
      raise ValueError.newException(
        "Incompatible line length in map " & file)
    result.width = width
    inc result.height

We split the file by line as well as by word. Each number is parsed to an unsigned integer and checked whether it is in the valid range of 0..255. The final width and height are calculated from the line length and number of lines. Errors in the map data cause an exception to be thrown, quitting our game.

We have to extend newGame to initialize the map now:

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer
  result.player = newPlayer(renderer.loadTexture("player.png"))
  result.map = newMap(renderer.loadTexture("grass.png"),
    "default.map")

At this point we have the texture and the tiles of the map. What’s missing is actually rendering all of this:

const
  tilesPerRow = 16
  tileSize: Point = (64.cint, 64.cint)

proc renderMap(renderer: RendererPtr, map: Map, camera: Vector2d) =
  var
    clip = rect(0, 0, tileSize.x, tileSize.y)
    dest = rect(0, 0, tileSize.x, tileSize.y)

  for i, tileNr in map.tiles:
    if tileNr == 0: continue

    clip.x = cint(tileNr mod tilesPerRow) * tileSize.x
    clip.y = cint(tileNr div tilesPerRow) * tileSize.y
    dest.x = cint(i mod map.width) * tileSize.x - camera.x.cint
    dest.y = cint(i div map.width) * tileSize.y - camera.y.cint

    renderer.copy(map.texture, unsafeAddr clip, unsafeAddr dest)

This is similar to renderTee but uses fixed size parts of the texture. The texture is cut into tiles of size 64 × 64 pixels with 16 tiles per line. We iterate over each tile in the map and render the tile tileNr from our map texture with. Tile 0 is the air tile and is always empty so we don’t need to render it, which improves performance as typical maps are in large parts empty.

Finally we have to render our map in the main proc after rendering the player, so that it is put on top of the player:

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)
  # Show the result on screen
  game.renderer.present()

For now we don’t have a moving camera, so we use the static game.camera to get a fixed rendering position. But for now we can’t even move, so we should finally do something with the user input and implement a simple physics model.

The end result of this section features our beautiful map rendering:

Full code for section 4

5. Physics and Collisions

For our game physics we decide to have 50 ticks per second. We will only calculate the next iteration of the game when a new tick has arrived, independent of whether our game runs at 60 fps or even 240 fps. Let’s add the ticks to our main proc:

import times

proc physics(game: Game) =
  discard

var
  startTime = epochTime()
  lastTick = 0

while not game.inputs[Input.quit]:
  game.handleInput()

  let newTick = int((epochTime() - startTime) * 50)
  for tick in lastTick+1 .. newTick:
    game.physics()
  lastTick = newTick

  renderer.render(game)

That’s our physics framework, but so far the physics does nothing. We can start by adding gravity:

proc physics(game: Game) =
  game.player.vel.y += 0.75
  game.player.pos += game.player.vel

Well, there goes our player. I guess we also need to be able to restart the player now, so that we can see this amazing animation again and again just by pressing r:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  game.player.vel.y += 0.75
  game.player.pos += game.player.vel

This looks exactly like the previous GIF on repeat, so you can imagine it or click on play a few times.

Moving left and right with a and d as well as jumping with space is now easy to implement:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  if game.inputs[Input.jump]:
    game.player.vel.y = -21

  let direction = float(game.inputs[Input.right].int -
                        game.inputs[Input.left].int)

  game.player.vel.y += 0.75
  game.player.vel.x = clamp(
    0.5 * game.player.vel.x + 4.0 * direction, -8, 8)

  game.player.pos += game.player.vel

The specific values are determined by trial and error. You can change them around if you have other preferences. Note that we don’t set the player position directly, instead we modify the velocity vector and add that to the position. This is important for collision detection.

While I tried to move around as if the walls and ground had an effect, I probably failed at deceiving you and you noticed that we still don’t have any collision detection and handling. This code is largely adapted from Teeworlds. It works by checking for horizontal and vertical collisions with a tile in moveBox, which manipulates the player’s position based on the passed velocity vector.

When a collision occurs the player is moved just out of the tile in the right direction. For the sake of simplicity in isSolid we consider every tile other than air, start, finish a solid block. Floating point player positions are converted to indices in the tile map.

import math

type
  Collision {.pure.} = enum x, y, corner

const
  playerSize = vector2d(64, 64)

  air = 0
  start = 78
  finish = 110

getTile reads a tile from a specified position of the map, making sure not to over- or underflow:

proc getTile(map: Map, x, y: int): uint8 =
  let
    nx = clamp(x div tileSize.x, 0, map.width - 1)
    ny = clamp(y div tileSize.y, 0, map.height - 1)
    pos = ny * map.width + nx

  map.tiles[pos]

isSolid determines whether the player can collide with a tile. As we said every tile other than air, start and finish are able to be collided with:

proc isSolid(map: Map, x, y: int): bool =
  map.getTile(x, y) notin {air, start, finish}

proc isSolid(map: Map, point: Point2d): bool =
  map.isSolid(point.x.round.int, point.y.round.int)

A player is on the ground when there is a block below either side of its feet:

proc onGround(map: Map, pos: Point2d, size: Vector2d): bool =
  let size = size * 0.5
  result =
    map.isSolid(point2d(pos.x - size.x, pos.y + size.y + 1)) or
    map.isSolid(point2d(pos.x + size.x, pos.y + size.y + 1))

Meanwhile testBox considers the player as an axis aligned boundary box, and tells us if the player is stuck inside of any solid walls:

proc testBox(map: Map, pos: Point2d, size: Vector2d): bool =
  let size = size * 0.5
  result =
    map.isSolid(point2d(pos.x - size.x, pos.y - size.y)) or
    map.isSolid(point2d(pos.x + size.x, pos.y - size.y)) or
    map.isSolid(point2d(pos.x - size.x, pos.y + size.y)) or
    map.isSolid(point2d(pos.x + size.x, pos.y + size.y))

moveBox now tries to apply the velocity vector vel to the player’s position pos. When this causes a collision the code tries to move the player only along the x axis, then only along the y axis, to find out which side of the tile the player collided with. If the player did not collide with any of the sides it hit a corner (that’s the real corner case!):

proc moveBox(map: Map, pos: var Point2d, vel: var Vector2d,
             size: Vector2d): set[Collision] {.discardable.} =
  let distance = vel.len
  let maximum = distance.int

  if distance < 0:
    return

  let fraction = 1.0 / float(maximum + 1)

  for i in 0 .. maximum:
    var newPos = pos + vel * fraction

    if map.testBox(newPos, size):
      var hit = false

      if map.testBox(point2d(pos.x, newPos.y), size):
        result.incl Collision.y
        newPos.y = pos.y
        vel.y = 0
        hit = true

      if map.testBox(point2d(newPos.x, pos.y), size):
        result.incl Collision.x
        newPos.x = pos.x
        vel.x = 0
        hit = true

      if not hit:
        result.incl Collision.corner
        newPos = pos
        vel = vector2d(0, 0)

    pos = newPos

The moveBox proc also returns a set of collisions, telling us what kind of collisions happened in this iteration. We don’t use that information, but it could be useful if you want to handle collisions in a special way instead of just pushing the player out of the collided wall.

Finally we can use onGround and moveBox:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  let ground = game.map.onGround(game.player.pos, playerSize)

  if game.inputs[Input.jump]:
    if ground:
      game.player.vel.y = -21

  let direction = float(game.inputs[Input.right].int -
                        game.inputs[Input.left].int)

  game.player.vel.y += 0.75
  if ground:
    game.player.vel.x = 0.5 * game.player.vel.x + 4.0 * direction
  else:
    game.player.vel.x = 0.95 * game.player.vel.x + 2.0 * direction
  game.player.vel.x = clamp(game.player.vel.x, -8, 8)

  game.map.moveBox(game.player.pos, game.player.vel, playerSize)

Jumping is only possible when standing on the ground. Horizontal movement in the air is calculated in a different way than on the ground, simulating different air and ground friction.

Full code for section 5

6. Camera

Did you see me disappear out of the window? Nice, right?! Well, actually that probably makes playing the rest of the map rather difficult. So while we can move nicely, the camera’s position is still fixed. In this section we will only make the camera move horizontally. With this information you can probably figure out how to make it move vertically as well if you want to. We only need to set the camera position when our player moved, right after calling game.physics():

const
  windowSize: Point = (1280.cint, 720.cint)

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  game.camera.x = game.player.pos.x - halfWin

for tick in lastTick+1 .. newTick:
  game.physics()
  game.moveCamera()

Now the camera always immediately follows the player’s horizontal position. Another approach for the camera is to have it follow the player only once the player leaves the center area of the screen, in this case 200 pixels:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  let
    leftArea  = game.player.pos.x - halfWin - 100
    rightArea = game.player.pos.x - halfWin + 100
  game.camera.x = clamp(game.camera.x, leftArea, rightArea)

An alternative approach is to have the camera follow the player fluidly. When the player is further away from the center the camera moves towards him faster. You can imagine the camera as being pulled by a rubber band connecting it to the player, the further away they get, the stronger the camera gets pulled to the player:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  let dist = game.camera.x - game.player.pos.x + halfWin
  game.camera.x -= 0.05 * dist

Choices are difficult so we can just implement all three and you can choose at compile time using -d:fluidCamera and -d:innerCamera:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  when defined(fluidCamera):
    let dist = game.camera.x - game.player.pos.x + halfWin
    game.camera.x -= 0.05 * dist
  elif defined(innerCamera):
    let
      leftArea  = game.player.pos.x - halfWin - 100
      rightArea = game.player.pos.x - halfWin + 100
    game.camera.x = clamp(game.camera.x, leftArea, rightArea)
  else:
    game.camera.x = game.player.pos.x - halfWin

Full code for section 6

7. Game State

Now that we can run around all of the map with the camera keeping up with us, we should give our game a purpose. You might have noticed the light and dark gray lines at the beginning and end of the map, suspiciously referred to as start and finish respectively. As you can probably guess we will use those as start and finish lines and record how quickly the player can get from start to finish:

type
  Time = ref object
    begin, finish, best: int

  Player = ref object
    texture: TexturePtr
    pos: Point2d
    vel: Vector2d
    time: Time

proc restartPlayer(player: Player) =
  player.pos = point2d(170, 500)
  player.vel = vector2d(0, 0)
  player.time.begin = -1
  player.time.finish = -1

proc newTime: Time =
  new result
  result.finish = -1
  result.best = -1

proc newPlayer(texture: TexturePtr): Player =
  new result
  result.texture = texture
  result.time = newTime()
  result.restartPlayer()

We’re now storing a Time object in the Player, telling us when the player began playing this round, how he finished last time and what his absolute best time is. By default the values are initialized to -1 to indicate an invalid value, otherwise they store ticks.

To format the time for display we use the excellent strfmt library’s string interpolation:

import strfmt

proc formatTime(ticks: int): string =
  let
    mins = (ticks div 50) div 60
    secs = (ticks div 50) mod 60
    cents = (ticks mod 50) * 2
  interp"${mins:02}:${secs:02}:${cents:02}"

Our game logic works as follows:

When the player walks through a start tile his time begins
When the player walks through a finish tile his finish time is set and printed to the terminal. If it is a new best time, so is his best time. The start time is reset.

proc getTile(map: Map, pos: Point2d): uint8 =
  map.getTile(pos.x.round.int, pos.y.round.int)

proc logic(game: Game, tick: int) =
  template time: untyped = game.player.time
  case game.map.getTile(game.player.pos)
  of start:
    time.begin = tick
  of finish:
    if time.begin >= 0:
      time.finish = tick - time.begin
      time.begin = -1
      if time.best < 0 or time.finish < time.best:
        time.best = time.finish
      echo "Finished in ", formatTime(time.finish)
  else: discard

We need to call the logic in our main proc of course:

for tick in lastTick+1 .. newTick:
  game.physics()
  game.moveCamera()
  game.logic(tick)

Now we can play through the map and finally get an output like this on the terminal:

Finished in 00:04:38

Full code for section 7

8. Text Rendering

It would be much nicer to have the text output with the result in the actual game window instead of on the terminal. For this we will now use SDL_ttf:

import sdl2/ttf

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y: cint, color: Color) =
  let surface = font.renderUtf8Solid(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  var source = rect(0, 0, surface.w, surface.h)
  var dest = rect(x, y, surface.w, surface.h)
  let texture = renderer.createTextureFromSurface(surface)

  sdlFailIf texture.isNil:
    "Could not create texture from rendered text"

  surface.freeSurface()

  renderer.copyEx(texture, source, dest, angle = 0.0, center = nil,
                  flip = SDL_FLIP_NONE)

  texture.destroy()

proc renderText(game: Game, text: string,
                x, y: cint, color: Color) =
  const outlineColor = color(0, 0, 0, 64)
  game.renderer.renderText(game.font, text, x, y, color)

What’s happening here is that we render a text with a FontPtr to an SDL2 surface (stored in RAM), which is then put into a texture (stored in GPU’s VRAM). This texture is then rendered to the screen at the defined position.

proc render(game: Game, tick: int) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)

  let time = game.player.time
  const white = color(255, 255, 255, 255)
  if time.begin >= 0:
    game.renderText(formatTime(tick - time.begin), 50, 100, white)
  elif time.finish >= 0:
    game.renderText("Finished in: " & formatTime(time.finish),
      50, 100, white)
  if time.best >= 0:
    game.renderText("Best time: " & formatTime(time.best),
      50, 150, white)

  # Show the result on screen
  game.renderer.present()

We need to initialize the TTF subsystem and now we also need to pass the current tick to render inside of our main function:

sdlFailIf(ttfInit() == SdlError): "SDL2 TTF initialization failed"
defer: ttfQuit()

...

game.render(lastTick)

As well as loading our DejaVuSans.ttf font inside of newGame:

result.font = openFont("DejaVuSans.ttf", 28)
sdlFailIf result.font.isNil: "Failed to load font"

The Game object now looks like this, including a font:

type
  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    font: FontPtr
    player: Player
    map: Map
    camera: Vector2d

This is what the text rendering looks like after finishing the map:

It noticeably doesn’t look all that great. The border of the text looks rugged. This happens because renderUtf8Solid does not use alpha blending, which is expensive. Instead every pixel is either entirely white or entirely transparent, never anything half-transparent in between. If we had a fixed background color for the text we could use renderUtf8Shaded, which takes a background color. If we want nicer output with dynamic backgrounds we can use renderUtf8Blended instead:

This looks better, but would be hard to see with a brighter background. We can draw an outline for our text to fix this, basically by drawing the text twice, once in half-transparent black and once in the proper color on top:

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y, outline: cint, color: Color) =
  font.setFontOutline(outline)
  let surface = font.renderUtf8Blended(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  var source = rect(0, 0, surface.w, surface.h)
  var dest = rect(x - outline, y - outline, surface.w, surface.h)
  ...

proc renderText(game: Game, text: string,
                x, y: cint, color: Color) =
  const outlineColor = color(0, 0, 0, 64)
  game.renderer.renderText(game.font, text, x, y, 2, outlineColor)
  game.renderer.renderText(game.font, text, x, y, 0, color)

Full code for section 8

9. Text Caching

If you looked at your CPU usage during this tutorial so far you might have noticed that the game needed nearly no CPU at all, about 3 % for 60 fps on my system. Once the texts are rendered this increases to 20% though. The reason for this is that right now we are regenerating the text textures every single frame, even if it didn’t change from the last frame. If you have fixed texts you can simply save the textures instead of recalculating them. But if you want more flexibility instead you can use a glyph or texture caching system. An example of such a system would be SDL_FontCache.

But instead we can write a little application-specific caching scheme in Nim. The heuristic we use is that mostly a single line in the code base will keep producing the same string, at least for some time. So we cache only a single text rendering for each line that prints something to the screen. That means we don’t have to do any lookups in a cache data structure and we only use a guaranteed constant amount of memory for caching:

type
  CacheLine = object
    texture: TexturePtr
    w, h: cint

  TextCache = ref object
    text: string
    cache: array[2, CacheLine]

proc newTextCache: TextCache =
  new result

A CacheLine is what we store, a pointer to a texture as well as the texture’s width and height. For our text rendering we need two of those as we render the text twice to get the outline effect. The text is also stored in TextCache to see if we already have the correct textures cached.

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y, outline: cint, color: Color): CacheLine =
  font.setFontOutline(outline)
  let surface = font.renderUtf8Blended(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  result.w = surface.w
  result.h = surface.h
  result.texture = renderer.createTextureFromSurface(surface)
  sdlFailIf result.texture.isNil: "Could not create texture from rendered text"

  surface.freeSurface()

We manipulated renderText to return a CacheLine that we can use:

proc renderText(game: Game, text: string, x, y: cint,
                color: Color, tc: TextCache) =
  let passes = [(color: color(0, 0, 0, 64), outline: 2.cint),
                (color: color, outline: 0.cint)]

  if text != tc.text:
    for i in 0..1:
      tc.cache[i].texture.destroy()
      tc.cache[i] = game.renderer.renderText(
        game.font, text, x, y, passes[i].outline, passes[i].color)
    tc.text = text

  for i in 0..1:
    var source = rect(0, 0, tc.cache[i].w, tc.cache[i].h)
    var dest = rect(x - passes[i].outline, y - passes[i].outline,
                    tc.cache[i].w, tc.cache[i].h)
    game.renderer.copyEx(tc.cache[i].texture, source, dest,
                         angle = 0.0, center = nil)

In two passes the text is rendered, if the text is cached from the cache directly, otherwise the old cache entry is removed and replaced with the new textures.

Nim’s metaprogramming allows us to use this seamlessly with a small template. A few days ago I wrote an article about Metaprogramming in Nim if you want to learn more about the powerful side of Nim.

template renderTextCached(game: Game, text: string,
                          x, y: cint, color: Color) =
  block:
    var tc {.global.} = newTextCache()
    game.renderText(text, x, y, color, tc)

proc render(game: Game, tick: int) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)

  let time = game.player.time
  const white = color(255, 255, 255, 255)
  if time.begin >= 0:
    game.renderTextCached(formatTime(tick - time.begin),
      50, 100, white)
  elif time.finish >= 0:
    game.renderTextCached("Finished in: " &
      formatTimeExact(time.finish), 50, 100, white)
  if time.best >= 0:
    game.renderTextCached("Best time: " &
      formatTimeExact(time.best), 50, 150, white)

  # Show the result on screen
  game.renderer.present()

Now each of the three renderTextCached calls get its own TextCache assigned, which is then used for the rest of the program execution. Note that this caching scheme only works under the assumption that there are relatively few separate lines of code that call renderTextCached and the ones that do often render the same text multiple times in a row. Good enough for our use case.

proc formatTime(ticks: int): string =
  let mins = (ticks div 50) div 60
  let secs = (ticks div 50) mod 60
  interp"${mins:02}:${secs:02}"

proc formatTimeExact(ticks: int): string =
  let cents = (ticks mod 50) * 2
  interp"${formatTime(ticks)}:${cents:02}"

We also reduced the exactness of the current time format because before it had to be recalculated every single frame. Now we’re back to using 4 % CPU and our final game looks like this:

Full code for section 9

Building

We can create a platformer.nimble file to tell Nimble, Nim’s package manager, how to build our package:

# Package

version       = "1.0"
author        = "Dennis Felsing"
description   = "An example platform game with SDL2"
license       = "MIT"

bin           = @["platformer"]

# Dependencies

requires "nim >= 0.10.0"
requires "sdl2 >= 1.1"
requires "strfmt >= 0.6"
requires "basic2d >= 0.1.0"

task tests, "Compile all tutorial steps":
  for i in 1..9:
    exec "nim c tutorial/platformer_part" & $i

All the intermediate code states from this article can be compiled with nimble tests. We can test the build by running nimble build, which creates a platformer binary that we can run. nimble install installs the same binary so that it is available in ~/.nimble/bin.

But once we run this installed platformer binary we get an error:

Error: unhandled exception: Could not load image player.png, SDL error: Couldn't open player.png [SDLException]

Actually this makes sense, because we load the files at runtime from the current directory, which can now be any directory. We have two choices for how to solve this:

Read in all files at compile time and get a binary that depends on no assets
Doing this is pretty simple in Nim as the compiler supports reading arbitrary files at compile time with staticRead:

import streams

template staticReadRW(filename: string): ptr RWops =
  const file = staticRead(filename)
  rwFromConstMem(file.cstring, file.len)

template staticReadStream(filename: string): string =
  const file = staticRead(filename)
  newStringStream(file)

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

  result.font = openFontRW(
    staticReadRW("DejaVuSans.ttf"), freesrc = 1, 28)
  sdlFailIf result.font.isNil: "Failed to load font"

  result.player = newPlayer(renderer.loadTexture_RW(
    staticReadRW("player.png"), freesrc = 1))
  result.map = newMap(renderer.loadTexture_RW(
    staticReadRW("grass.png"), freesrc = 1),
    staticReadStream("default.map"))

Define a directory where we store our data assets to load them at runtime. This enables the player to switch them out for custom ones without the need to recompile the binary. And as we’ve seen with the DDNet Skin Database this might be a useful feature. We find the data directory by looking where in the file system our binary is. Let’s implement this and make the old embedded compile time assets optional:

import os, streams

const dataDir = "data"

when defined(embedData):
  template readRW(filename: string): ptr RWops =
    const file = staticRead(dataDir / filename)
    rwFromConstMem(file.cstring, file.len)

  template readStream(filename: string): Stream =
    const file = staticRead(dataDir / filename)
    newStringStream(file)
else:
  let fullDataDir = getAppDir() / dataDir

  template readRW(filename: string): ptr RWops =
    var rw = rwFromFile(cstring(fullDataDir / filename), "r")
    sdlFailIf rw.isNil: "Cannot create RWops from file"
    rw

  template readStream(filename: string): Stream =
    var stream = newFileStream(fullDataDir / filename)
    if stream.isNil: raise ValueError.newException(
      "Cannot open file stream:" & fullDataDir / filename)
    stream

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

  result.font = openFontRW(
    readRW("DejaVuSans.ttf"), freesrc = 1, 28)
  sdlFailIf result.font.isNil: "Failed to load font"

  result.player = newPlayer(renderer.loadTexture_RW(
    readRW("player.png"), freesrc = 1))
  result.map = newMap(renderer.loadTexture_RW(
    readRW("grass.png"), freesrc = 1),
    readStream("default.map"))

This also requires us to change newMap to accept a Stream instead of a filename:

proc newMap(texture: TexturePtr, map: Stream): Map =
  new result
  result.texture = texture
  result.tiles = @[]

  var line = ""
  while map.readLine(line):
    ...

When compiling you can set -d:embedData like this:

$ nim -d:release c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 129K Jun 13 14:54 platformer*
$ nim -d:release -d:embedData c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 888K Jun 13 14:55 platformer*

You can find our final platform game code in the repository.

Now we can submit the repository to the Nimble packages as a pull request and soon all Nim developers can install the package right from their terminal with a nimble install platformer and play by just running platformer.

A circle.yml file defines how to run and compile our repository and is used to make sure it keeps building fine whenever changes are made.

Binary Distribution

But Nim developers are probably not the main target group of our game, so ideally we we also want to be able to build binaries for a few common platforms, Linux x86, x86-64 and Windows x86, x86-64 for us. Building for Mac OS X is a bit more involved, but you can check out how DDNet does it.

Of course we could just set up a VM for each system that we want to build for and use the instructions from the start of this article. But that’s tedious and we want the convenience of building on a single machine.

Linux

Note that I’m on Arch Linux, but this should be possible in an analogous way on other Linux distributions:

Building a portable binary for Linux is a pain because of glibc. When you compile on a system with a newer glibc version it might not run on a system with an older one. A common solution is to use a build system with an old Linux install. Alternatively an old Debian chroot can be created with debootstrap. There is also Linux Standard Base which aims to solve this problem, but I haven’t used yet.

A more hacky solution is to create the binary on the new system and check what symbols exactly are linked against the newer glibc version. In our case I want everything to use GLIBC_2.2.5, so I check for anything else:

objdump -T platformer | grep GLIBC | grep -v 2.2.5
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy

Okay, only memcpy is the problem. We can force the linker to use an old version of memcpy and another common problem, realpath, like this in C code with inline assembler:

__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");

But this would require for us to insert this into every C file that we generate. Or we abuse the nimbase.h file and insert it there and compile with:

$ head -n2 glibc-hack/nimbase.h
__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");
$ nim -d:release --passC:-Iglibc-hack c platformer
$ objdump -T platformer | grep memcpy
0000000000000000      DF *UND* 0000000000000000  GLIBC_2.2.5 memcpy
$ objdump -T platformer|grep GLIBC|grep -v 2.2.5

Now our binary works on Linux versions using glibc 2.2.5 or newer. Note that the user still needs SDL2, SDL_image2 and SDL_ttf2 installed.

When you’re linking dynamically and want to distribute shared libraries with your binary you can compile with nim --passC:-Wl,-rpath,. c platformer and put the shared libraries into the same directory as the binary.

At least building for x86 is easy on Linux as long as you’re on x86-64 and have gcc-multilib installed:

$ yaourt -S lib32-sdl2 lib32-sdl2_image lib32-sdl2_ttf
$ nim --cpu:i386 --passC:-m32 --passL:-m32 -d:release c platformer

Windows

Surprisingly it is much easier to build portable binaries for Windows, even from Linux:

$ pacman -S mingw-w64-gcc
$ nim --os:windows --cpu:amd64 --gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc -d:release c platformer
$ nim --os:windows --cpu:i386 --gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc -d:release c platformer

SDL libraries for Windows can be downloaded from the SDL2 website (image, ttf).

Stripping the binaries of symbols with strip -s platformer is a good idea if you want to save some space and don’t care about debugging your binary.

Automated Build Script

With all this information we can now write a fully automated release build script, written in Nim as well:

import os, strfmt

const
  app = "platformer"
  version = "1.0"

  builds = [
    (name: "linux_x86", os: "linux", cpu: "i386",
     args: "--passC:-m32 --passL:-m32"),
    (name: "linux_x86_64", os: "linux", cpu: "amd64",
     args: "--passC:-Iglibc-hack"),
    (name: "win32", os: "windows", cpu: "i386",
     args: "--gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc"),
    (name: "win64", os: "windows", cpu: "amd64",
     args: "--gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc"),
  ]

removeDir "builds"

for name, os, cpu, args in builds.items:
  let
    dirName = app & "_" & version & "_" & name
    dir = "builds" / dirName
    exeExt = if os == "windows": ".exe" else: ""
    bin = dir / app & exeExt

  createDir dir
  if execShellCmd(interp"nim --cpu:${cpu} --os:${os} ${args} -d:release -o:${bin} c ${app}") != 0: quit 1
  if execShellCmd(interp"strip -s ${bin}") != 0: quit 1
  copyDir("data", dir / "data")
  if os == "windows": copyDir("libs" / name, dir)
  setCurrentDir "builds"
  if os == "windows":
    if execShellCmd(interp"zip -9r ${dirName}.zip ${dirName}") != 0: quit 1
  else:
    if execShellCmd(interp"tar cfz ${dirName}.tar.gz ${dirName}") != 0: quit 1
  setCurrentDir ".."

Linux users have to install sdl2, sdl2_image, sdl2_ttf using their package manager. Windows users get them bundled. Our build script creates this directory structure when we run it with nim -r c release:

.
├── platformer_1.0_linux_x86
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   └── platformer
├── platformer_1.0_linux_x86.tar.gz
├── platformer_1.0_linux_x86_64
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   └── platformer
├── platformer_1.0_linux_x86_64.tar.gz
├── platformer_1.0_win32
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   ├── libfreetype-6.dll
│   ├── libpng16-16.dll
│   ├── platformer.exe
│   ├── SDL2.dll
│   ├── SDL2_image.dll
│   ├── SDL2_ttf.dll
│   └── zlib1.dll
├── platformer_1.0_win32.zip
├── platformer_1.0_win64
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   ├── libfreetype-6.dll
│   ├── libpng16-16.dll
│   ├── platformer.exe
│   ├── SDL2.dll
│   ├── SDL2_image.dll
│   ├── SDL2_ttf.dll
│   └── zlib1.dll
└── platformer_1.0_win64.zip

Resulting downloads here: Win64, Win32, Linux x86_64, Linux x86

Final Words

Somehow my articles are getting longer and this journey towards writing a simple platform game took a few more explanations than expected. I hope this article wasn’t too long and was instructive and helpful to get an understanding of how to write platform games in Nim with SDL2.

All the material for this article is available in the repository on GitHub. Because I changed around things late into the article I might have made a mistake or missed something. If you find a bug or have a comment you can drop me an email at dennis@felsing.org.

Discussions on r/programming and Hacker News.

Introduction to Metaprogramming in Nim

2016-06-06T00:00:00-04:00

Introduction to the Introduction (Meta-Introduction)

Wikipedia gives us a nice description of metaprogramming:

Metaprogramming is the writing of computer programs with the ability to treat programs as their data. It means that a program could be designed to read, generate, analyse and/or transform other programs, and even modify itself while running.

In this article we will explore Nim’s metaprogramming capabilities, which are quite powerful and yet still easy to use. After all great metaprogramming is one of Nim’s main features. The general rule is to use the least powerful construct that is still powerful enough to solve a problem, in this order:

Normal procs and inline iterators
Generic procs and closure iterators
Templates
Macros

So before looking at Nim’s two main metaprogramming constructs, templates and macros, we’ll look at what we can do with procs and iterators as well.

Regular Programming Constructs

Normal procs

We’re in normal programming land here. Regular procedures are what you know as functions elsewhere and they’re pretty easy to define and use:

proc sayHi(name: string) =
  echo "Hello ", name

sayHi("World")
sayHi "World"
"World".sayHi

Generic procs

With generics we can define procs that work on multiple types. Actually a new proc will be generated based on our generic definition for each instantiation:

proc min[T](x, y: T): T =
  if x < y: x else: y

echo min(2, 3) # more explicitly: min[int](2, 3)
echo min("foo", "bar") # min[string]("foo", "bar")

Inline iterators

Inline iterators are the default iterators in Nim. They get compiled into high performance loops:

iterator reverseItems(x: string): char =
  for i in countdown(x.high, x.low):
    yield x[i]

for c in "foo".reverseItems:
  echo c

So this code gets compiled into:

let x = "foo"
for c in countdown(x.high, x.low):
  let c = x[i]
  echo c

Of course we can make iterators generic too:

iterator reverseItems[T](x: T): auto =
  for i in countdown(x.high, x.low):
    yield x[i]

Closure iterators

Inline iterators simultaneously have the advantage and disadvantage of being translated into loops. This means you can not pass them around. This limitation can be lifted by using closure iterators instead:

import math

proc powers(m: int): auto =
  #return iterator: int {.closure.} = # Make a closure explicitly
  return iterator: int = # Compiler makes this a closure for us
    for n in 0..int.high:
      yield n^m

var
  squares = powers(2)
  cubes = powers(3)

for i in 1..4:
  echo "Square: ", squares() # 0, 1, 4, 9
for i in 1..4:
  echo "Cube: ", cubes()     # 0, 1, 8, 27
echo "Square: ", squares()   # 16
echo "Cube: ", cubes()       # 64

for x in squares(): # Go through all the remaining squares
  echo "Square: ", x         # 25, 36, 49, 64, ...

As you can see closure iterators keep their state. You can call them again and get the next value, or use them inside of a for-loop to get out many values.

Templates

You can think of templates as Nim’s equivalent to the C preprocessor. But templates are written in Nim itself and fit well into the rest of the language.

Templates simply insert their code at the invocation site, working at the level of the abstract syntax tree. They can be used in just the same way as procs.

Logger

A common example are loggers, which we looked at in another article already. Consider that you want to have extensive debug logging in your program. A trivial implementation would look like this:

import strutils, times, os

type Level* {.pure.} = enum
  debug, info, warn, error, fatal

var logLevel* = Level.debug

proc debug*(args: varargs[string, `$`]) =
  if logLevel <= Level.debug:
    echo "[$# $#]: $#" % [getDateStr(), getClockStr(),
      join args]

proc expensiveDebuggingInfo*: string =
  sleep(milsecs = 1000)
  result = "Everything looking good!"

debug expensiveDebuggingInfo()

[2016-06-05 22:00:50]: Everything looking good!

We have to call expensiveDebuggingInfo to get the debugging info, which is fine right now since our logLevel is set to Level.debug. But it stops being fine when we instead set logLevel to anything higher than debug. Then it still takes a full second to evaluate the expensiveDebuggingInfo parameter for debug, but inside of debug nothing is done with that information. This is of course a consequence of call-by-value argument evaluation, which Nim uses, just as most other languages do. A notable exception would be lazy evaluation in Haskell, where this kind of logger would work perfectly fine, only calling expensiveDebuggingInfo when its value is actually needed.

But let’s stay in Nim-land and use a template instead of a proc to magically fix this:

template debug*(args: varargs[string, `$`]) =
  if logLevel <= Level.debug:
    const module = instantiationInfo().filename[0 .. ^5]
    echo "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
      module, join args]

[2016-06-05 22:01:30][logger]: Everything looking good!

Note that we also conveniently use instantiationInfo() to find out at what location in the program our template was instantiated, something we could not do using a procedure.

We can still call the template in the exact same way as the proc. But now we have the advantage that the template is inlined at compiletime, so expensiveDebuggingInfo is only called if the runtime logLevel actually requires it. Perfect.

Safe locking

Another problem that can be solved with a template is automatically acquiring and releasing a system lock:

import locks

template withLock(lock: Lock, body: untyped) =
  acquire lock
  try:
    body
  finally:
    release lock

Compile with --threads:on for platform independent lock support.

This looks pretty simple, we just acquire the lock, execute the passed statements and finally release the lock, even if exceptions have been thrown. We can pass any set of statements as the body. The usage is as easy as using a built-in if statement:

var lock: Lock
initLock lock

withLock lock:
  echo "Do something that requires locking"
  echo "This might throw an exception"

When our template accepts a value of type stmt we can use the colon to pass an entire indented block of code. When we have multiple parameters of type stmt the do notation can be used.

This gets transformed into:

var lock: Lock
initLock lock

acquire lock
try:
  echo "Do something that requires locking"
  echo "This might throw an exception"
finally:
  release lock

Now we will never forget to call release lock. You could use this to make a higher level locking library that only exposes withLock instead of the lower-level acquire and release primitives.

Macros

Just like templates, macros are executed at compiletime. But with templates you can only do constant substitutions in the AST. With macros you can analyze the passed arguments and create a new AST at the current position in any way you want. A nice property of Nim is that these compiletime macros are also written in the regular Nim language, so there is no need to learn another language.

A simple way to create an AST is to use parseStmt and parseExpr to parse the regular textual representation into a NimNode. For example parseStmt("result = 10") returns this AST:

StmtList
  Asgn
    Ident !"result"
    IntLit 10

A very useful way to find the AST of a piece of code is dumpTree:

import macros

dumpTree:
  result = 10

This is the same output as you get with treeRepr:

import macros

static:
  echo treeRepr(parseStmt("result = 10"))

Alternatively you can use lispRepr to get a lisp-like representation:

StmtList(Asgn(Ident(!"result"), IntLit(10)))

Finally there is also the repr proc, which turns a NimNode AST back into its textual representation.

Many beginners start by piecing strings together and finally calling parseStmt on them. While this works it is inefficient and prone to bugs. Instead you can use the macros module to create NimNodes of all kinds yourself. dumpTree gives you a hint if you’re not sure how a specific piece of code will look in its AST representation.

JSON Parsing

JSON is pretty popular, so let’s improve the support for it in Nim. What we want is to have a magical %* so that we can write JSON directly in Nim source code and have it checked at compile time, like this:

var j1 = %*
  [
    {
      "name": "John",
      "age": 30
    },
    {
      "name": "Susan",
      "age": 31
    }
  ]

So far if you want to use JSON in Nim, you have to use the JSON constructor % a lot:

import json

var j2 =
  %[
    %{
      "name": %"John",
      "age": %30
    },
    %{
      "name": %"Susan",
      "age": %31
    }
  ]

Looks annoying. How can we implement %*? As a macro of course!:

macro `%*`*(x: expr): expr = toJson(x)

Ok, that doesn’t do anything interesting yet. We just call the still unspecified compile time proc toJson and return the result. We want toJson to traverse the passed AST x and create a new AST, which inserts a % call at just the right places, exactly as it would happen if we added the % calls manually.

For this purpose we print the AST of j2 by putting it into dumpTree from the macros module:

import json, macros

dumpTree:
  %[
    %{
      "name": %"John",
      "age": %30
    },
    %{
      "name": %"Susan",
      "age": %31
    }
  ]

We get the following AST printed when compiling this program:

Prefix
  Ident !"%"
  Bracket
    Prefix
      Ident !"%"
      TableConstr
        ExprColonExpr
          StrLit name
          Prefix
            Ident !"%"
            StrLit John
        ExprColonExpr
          StrLit age
          Prefix
            Ident !"%"
            IntLit 30
    Prefix
      Ident !"%"
      TableConstr
        ExprColonExpr
          StrLit name
          Prefix
            Ident !"%"
            StrLit Susan
        ExprColonExpr
          StrLit age
          Prefix
            Ident !"%"
            IntLit 31

This turned out quite big, but from here we can see how the AST we want to construct looks like. We do the same for j1 to see what we’re working with:

StmtList
  Bracket
    TableConstr
      ExprColonExpr
        StrLit name
        StrLit John
      ExprColonExpr
        StrLit age
        IntLit 30
    TableConstr
      ExprColonExpr
        StrLit name
        StrLit Susan
      ExprColonExpr
        StrLit age
        IntLit 31

The idea now is to insert a % at each level, except in front of the "name" and "age" in our case, the first elements in colon expressions.

proc toJson(x: PNimrodNode): PNimrodNode {.compiletime.} =
  case x.kind
  of nnkBracket: # Corresponds to Bracket in dumpTree
    result = newNimNode(nnkBracket)
    for i in 0 .. <x.len:
      result.add(toJson(x[i])) # Recurse to add %

  of nnkTableConstr: # nnk stands for Nim node kind
    result = newNimNode(nnkTableConstr)
    for i in 0 .. <x.len:
      assert x[i].kind == nnkExprColonExpr
      result.add(newNimNode(nnkExprColonExpr)
        .add(x[i][0])           # First element: no %
        .add(toJson(x[i][1]))): # Second element: Recurse to add %

  else:
    result = x # End of recursion

  result = result.prefix("%") # Surround this level with %

And that’s it! Now our %* works just as we want it to. If we did anything wrong, we can modify the macro to check the actual code it produces:

macro `%*`*(x: expr): expr =
  result = toJson(x)
  echo result.repr # Print code representation of AST

This prints:

% [% {"name": % "John", "age": % 30}, % {"name": % "Susan", "age": % 31}]

Perfect! This macro we just developed landed in Nim’s json module already.

Enum Parsing optimization

With enums we can create new types that contain ordered values, just like this:

  type Fruit = enum Apple, Banana, Cherry

Strings can be parsed to an enum using parseEnum from strutils:

  let fruit = parseEnum[Fruit]("cherry")

If we do this a lot, we notice that it’s kind of slow though:

for i in 1 .. 10_000_000:
  var select = parseEnum[Fruit]("cherry")
  doAssert select == Cherry

This takes 2.2 seconds on my machine. Let’s look at the definition of parseEnum to find out why:

proc parseEnum*[T: enum](s: string): T =
  ## Parses an enum ``T``.
  ##
  ## Raises ``ValueError`` for an invalid value in `s`. The
  ## comparison is done in a style insensitive way.
  for e in low(T)..high(T):
    if cmpIgnoreStyle(s, $e) == 0:
      return e
  raise newException(ValueError, "invalid enum value: " & s)

We can see the problem already. We iterate through all the values inside the enum type, from low(T) to high(T). Then $e creates a string of each enum value, which is quite expensive. Since we already know the type of the enum at compile time, we could create the strings at compile time as well.

Again, let’s think about what we want the result to look like before writing the macro. Basically what we want to do is unroll the for loop at compile time:

if cmpIgnoreStyle(s, "Apple") == 0: return Apple
if cmpIgnoreStyle(s, "Banana") == 0: return Banana
if cmpIgnoreStyle(s, "Cherry") == 0: return Cherry
raise newException(ValueError, "invalid enum value: " & s)

Now we can create the proc. Other than in the last example we won’t create the AST manually this time. Instead we use parseStmt to create a statement AST from a string containing Nim code. An equivalent parseExpr for expressions exists as well. Here’s how the final proc with a macro inside looks:

proc parseEnum*[T: enum](s: string): T =
  macro m: untyped =
    result = newStmtList()
    for e in T: result.add parseStmt(
      "if cmpIgnoreStyle(s, \"$1\") == 0: return $1".format(e))

    result.add parseStmt(
      "raise newException(ValueError, \"invalid enum value: \"&s)")

    #echo result.repr # To make sure we get what we want

  m() # Actually invoke the macro to insert the statements here

Running the same code with our new implementation of parseEnum takes 0.5 seconds now, about 4 times faster than before. Great!

HTML DSL

We can use Nim’s templates and macros to create domain specific languages (DSL) that are translated into Nim code at compiletime. Nim’s syntax is quite flexible, so this is a powerful tool. As an example we build a simple HTML DSL.

The goal is to be able to write this:

proc page(title, content: string) {.htmlTemplate.} =
  html:
    head:
      title: title
    body:
      h1: title
      p: "Default Content"
      p: content

echo page("My own website", "My extra content")

And thus print the following HTML:

<html>
  <head>
    <title>
      My own website
    </title>
  </head>
  <body>
    <h1>
      My own website
    </h1>
    <p>
      Default Content
    </p>
    <p>
      My extra content
    </p>
  </body>
</html>

For convenience we want to use the htmlTemplate macro as a pragma, annotated as {.htmlTemplate.}. Instead we could also write it in this way:

htmlTemplate:
  proc page(title, content: string) =
    html:
      head:
        title: title
      body:
        h1: title
        p: "Default Content"
        p: content

The htmlTemplate macro shall transform the page proc, adding a string return type and creating a new body out of the DSL definition, into this:

proc page(title, content: string): string =
  result = ""
  result.add "<html>\n"
  ...
  result.add "</html>\n"

Looks simple enough, here’s how the macro works:

macro htmlTemplate(procDef: expr): untyped =
  procDef.expectKind nnkProcDef

  # Same name as specified
  let name = procDef[0]

  # Return type: string
  var params = @[newIdentNode("string")]
  # Same parameters as specified
  for i in 1..<procDef[3].len:
    params.add procDef[3][i]

  var body = newStmtList()
  # result = ""
  body.add newAssignment(newIdentNode("result"),
    newStrLitNode(""))
  # Recurse over DSL definition
  body.add htmlInner(procDef[6])

  # Return a new proc
  result = newStmtList(newProc(name, params, body))

The real magic of recursively handling the HTML tags happens in htmlInner of course, a compiletime proc that calls itself recursively to iterate over the body definition:

template write(arg: expr) =
  result.add newCall("add", newIdentNode("result"), arg)

template writeLit(args: varargs[string, `$`]) =
  write newStrLitNode(args.join)

proc htmlInner(x: NimNode, indent = 0): NimNode
              {.compiletime.} =
  x.expectKind nnkStmtList
  result = newStmtList()
  let spaces = repeat(' ', indent)
  for y in x:
    case y.kind
    of nnkCall:
      y.expectLen 2
      let tag = y[0]
      tag.expectKind nnkIdent
      writeLit spaces, "<", tag, ">\n"
      # Recurse over child
      result.add htmlInner(y[1], indent + 2)
      writeLit spaces, "</", tag, ">\n"
    else:
      writeLit spaces
      write y
      writeLit "\n"

We can check that we get the expected output by adding a simple echo result.repr at the end of htmlTemplate:

proc page(title, content: string): string =
  result = ""
  add(result, "<html>\x0A")
  add(result, "  <head>\x0A")
  ...
  add(result, "</html>\x0A")

Where \x0A is just the newline character. Looks good and the output works!

emerald is a much more complete HTML DSL that works in a similar manner. A simpler HTML generator is included in the standard library in the htmlgen module.

Conclusion

I hope you enjoyed this trip through Nim’s metaprogramming capabilities. Always remember: With great power comes great responsibility, so use the least powerful construct that does the job. This reduces complexity and makes it easier to understand the code and keep it maintainable.

For further information and reference see:

Discuss on Hacker News and r/programming.

DDNet Server Statistics with ServerStatus, RRDtool and Nim

2016-05-14T00:00:00-04:00

About a month ago I set up statistics for the official DDNet servers. My motivations for this are:

Monitor the servers more easily
Get notified about server problems
Have nice graphs to look at

The choices for the software used are mainly made to keep resource usage low, a general principle used for DDNet since we run on cheap VPSes all around the world and are limited in CPU and memory resources. In the rest of this post we will explore the 3 major tools used, their purpose in our solution as well as their performance impact:

ServerStatus: Gather live server statistics
RRDtool: Record and graph data
Nim: Favorite programming language for performance and readability

Gathering live server statistics with ServerStatus

We’ve been running BotoX’s ServerStatus to get live server statistics for some time now. It works quite well to quickly notice major server problems like a high load or incoming (D)DoS attacks, provided that you keep an eye out for it.

We use ServerStatus by running its simple Python client on each server to gather interesting information. The client transmits that data by TCP to the C/C++ server, which aggregates it into a JSON file. This JSON file is then fetched and displayed every two seconds by the JavaScript frontend of our Status page.

On a regular Saturday morning the end result looks like this:

On a quick glance you notice that the DDNet.tw server has high CPU usage (which is totally normal since it runs some hefty cron jobs every 20 minutes) and DDNet RUS is receiving a small DoS attack with just 1.6 MB/s (unfortunately also totally normal). Apart from that everything looks fine.

ServerStatus footprint, calculated from Linux /proc statistics as follows (in Nim):

import os, strutils, posix, strfmt
# CPU and memory usage of process, based on PROC(5) and
# http://stackoverflow.com/a/16736599
let
  pid = paramStr(1)
  # Clock ticks per second
  frequency = sysconf(SC_CLK_TCK)
  # Size of memory page in bytes
  pagesize  = sysconf(SC_PAGESIZE)

  uptime    = readFile("/proc/uptime").split[0].parseFloat
  fields    = readFile("/proc/" & pid & "/stat").split

  # Amount of time in user mode
  utime     = fields[13].parseInt
  # Amount of time in kernel mode
  stime     = fields[14].parseInt
  # Amount of children time in user mode
  cutime    = fields[15].parseInt
  # Amount of children time in kernel mode
  cstime    = fields[16].parseInt
  # Time process started after boot
  starttime = fields[21].parseInt

  # Resident Set Size: number of pages in memory
  rssmem    = fields[23].parseInt

  totaltime = utime + stime + cutime + cstime
  seconds   = uptime - (starttime / frequency)

  cpuusage  = 100 * (totaltime / frequency) / seconds
  memusage  = rssmem * pagesize / 1_000_000

echo interp"${cpuusage:.2f} % CPU ${memusage:.2f} MB Memory"

Part	CPU	Memory
Client	0.14 %	3.70 MB
Server (9 clients)	0.58 %	0.18 MB

Recording and graphing data with RRDtool

I haven’t used RRDtool for about 7 years, but it’s still an excellent tool to record data into a fixed-size round robin database. For us three functions of RRDtool are important: create to create the database, update to add a new value to be aggregated into the database, and graph to render the database into a beautiful graph.

CPU, network and memory are the most important resources for me, so their usage should be recorded. Let us use network traffic as an example and create a database:

First we need to think about what data we want to record in the RRD:

sample = 30 seconds

day  =   2880 samples =   6 * 480 pixels, each pixel is 03:00 min
days =  20160 samples =  42 * 480 pixels, each pixel is 21:00 min
days = 141120 samples = 147 * 960 pixels, each pixel is 73:30 min

Then we can use this to create the actual database file:

rrdtool create ddnet.tw-net.rrd `# File name` \
  --step 30 `# Interval in seconds with which data is fed` \
  DS:network_rx:GAUGE:60:0:U `# Data source receiving` \
  DS:network_tx:GAUGE:60:0:U `# DS Sending` \
  RRA:AVERAGE:0.5:6:480 `# Round robin archive for 1 day` \
  RRA:AVERAGE:0.5:42:480 `# RRA for 7 days` \
  RRA:AVERAGE:0.5:147:960 `# RRA for 49 days`

If you’re curious about what exactly happens here, you can find more information in rrdcreate(1).

The resulting ddnet.tw-net.rrd file is just 32 KB in size and will forever stay that exact size. (All our databases together are just 1 MB.) New data in each round robin archive simply overwrites the oldest data. A disadvantage of RRDtool is that you need to think ahead and plan what data you want to store.

The next step is to put new data into our little database, which we should do every 30 seconds:

rrdtool update ddnet.tw-net.rrd N:42:1234

Super simple! 42 is our network_rx value, 1234 the value for network_tx. These values are now aggregated using the AVERAGE and finally put into their respective archives.

Once we have enough values we can finally create the graph, for example for 1 day:

rrdtool graph ddnet.tw-net-1d.png --rigid --base 1000 \
  --width 419 --height 150 --logarithmic --units=si -a PNG \
  `# Calculation over last day only` \
  --vertical-label "Bytes/s" --start now-1d \
  `# Fetch data from RRD file` \
  DEF:network_rx=ddnet.tw-net.rrd:network_rx:AVERAGE \
  DEF:network_tx=ddnet.tw-net.rrd:network_tx:AVERAGE \
  `# Calculate aggregates based on data` \
  VDEF:network_rx_a=network_rx,AVERAGE \
  VDEF:network_rx_m=network_rx,MAXIMUM \
  VDEF:network_rx_c=network_rx,LAST \
  VDEF:network_rx_s=network_rx,TOTAL \
  VDEF:network_tx_a=network_tx,AVERAGE \
  VDEF:network_tx_m=network_tx,MAXIMUM \
  VDEF:network_tx_c=network_tx,LAST \
  VDEF:network_tx_s=network_tx,TOTAL \
  `# Draw area graph in light colors` \
  AREA:network_tx#fee8c8: \
  AREA:network_rx#e0e0e0: \
  `# Draw clear area outline on top` \
  LINE1:network_tx#e34a33:"out" \
  `# Print aggregate values to legend` \
  GPRINT:network_tx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_tx_m:"max\: %6.2lf %sB" \
  GPRINT:network_tx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_tx_s:"sum\: %6.2lf %sB\n" \
  `# Other area outline` \
  LINE1:network_rx#636363:"in " \
  GPRINT:network_rx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_rx_m:"max\: %6.2lf %sB" \
  GPRINT:network_rx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_rx_s:"sum\: %6.2lf %sB\n"

As always, the manual of rrdgraph explains the possibilities.

I’m using RRDtool 1.6.0 instead of 1.4.8 because I very much prefer its density of x-axis labels. Here are the outputs of our database:

RRDtool 1.4.8: RRDtool 1.6.0:

RRDtool footprint:

Part	Runtime	Memory
Create (9 servers)	0.01 s	1.74 MB
Graph (9 servers)	1.99 s	2.82 MB

Putting it together with Nim

To aggregate the raw data from ServerStatus into 30-second packets I use a small Nim program. It automatically creates new databases when a new server is added and keeps them updated:

import common, json, osproc, os, times, strutils, tables

type
  Data = object
    network_rx, network_tx: BiggestInt
    cpu, memory_used, memory_total, swap_used, swap_total: BiggestInt
    load: float

const freq = 30 # report new data to rrd every 30 seconds

var
  lastUpdated: BiggestInt = 0
  dataTable = initTable[string, Data]()
  count = 0
  countTable = initTable[string, int]()

proc rrdCreate(file, dataSources: string) =
  discard execCmd(rrdtool & " create " & file & " --step " & $freq & " " & dataSources &
    " RRA:AVERAGE:0.5:6:480 RRA:AVERAGE:0.5:42:480 RRA:AVERAGE:0.5:147:960")

proc rrdUpdate(file: string, values: varargs[string, `$`]) =
  var valuesString = ""
  for value in values:
    if valuesString.len > 0:
      valuesString.add ":"
    valuesString.add value
  discard execCmd(rrdtool & " update " & file & " N:" & valuesString)

proc updateServer(server: JsonNode) =
  let domain = server["type"].str

  for name, value in dataTable.mgetOrPut(domain, Data()).fieldPairs:
    when value is BiggestInt:
      value += server[name].num
    elif value is float:
      value += server[name].fnum
    else:
      error "Unhandled type in Data object"

  inc countTable.mgetOrPut(domain, 0)

  # Only save data if we got 30 values in the expected time span
  if count == freq and countTable[domain] == freq:
    let data = dataTable[domain]
    if data == Data():
      dataTable.del(domain)
      return
    else:
      dataTable[domain] = Data()

    let
      fileNet = (rrdDir / domain) & "-net.rrd"
      fileCpu = (rrdDir / domain) & "-cpu.rrd"
      fileMem = (rrdDir / domain) & "-mem.rrd"

    if not existsFile fileNet:
      fileNet.rrdCreate("DS:network_rx:GAUGE:60:0:U DS:network_tx:GAUGE:60:0:U")
    if not existsFile fileCpu:
      fileCpu.rrdCreate("DS:cpu:GAUGE:60:0:100 DS:load:GAUGE:60:0:U")
    if not existsFile fileMem:
      filemem.rrdCreate("DS:memory_used:GAUGE:60:0:U DS:memory_total:GAUGE:60:0:U DS:swap_used:GAUGE:60:0:U DS:swap_total:GAUGE:60:0:U")

    fileNet.rrdUpdate(data.network_rx div freq, data.network_tx div freq)
    fileCpu.rrdUpdate(min(data.cpu div freq, 100), data.load / freq)
    fileMem.rrdUpdate(data.memory_used div freq, data.memory_total div freq, data.swap_used div freq, data.swap_total div freq)

proc updateAllServers =
  let statsJson = parseFile statsJsonFile
  let newUpdated = parseBiggestInt statsJson["updated"].str

  if newUpdated <= lastUpdated:
    return

  inc count

  for server in statsJson["servers"]:
    try:
      updateServer(server)
    except:
      discard

  if count == freq:
    count = 0
    for val in countTable.mvalues:
      val = 0

while true:
  let startTime = epochTime()

  updateAllServers()

  sleep(int(epochTime() - startTime + 1) * 1000) # every second

The final graphs can be seen on the DDNet Server Statistics page.

Nim monitor footprint:

Part	CPU	Memory
Monitor (9 servers)	0.03 %	0.89 MB

Alerts through Cron and Mail

Now we certainly have nice graphs, but automated alerts about suspicious events would be even better, for example:

Network traffic over 2 MB/s for 4 min
Memory and swap over 90% for 4 min
CPU over 90% for 21 min
Load over 10 for 21 min
Server unreachable for 1 hour

To check these conditions a cron job is run regularly and thanks to a MAILTO entry a mail is sent when an alert has been triggered.

We can get out a single value from the database to standard output using PRINT:

rrdtool graph x -s-4min \
  DEF:v=ddnet.tw-net.rrd:network_rx:AVERAGE \
  VDEF:vm=v,AVERAGE \
  PRINT:vm:%lf

The alert program itself is written in Nim as well and merely gets a few values from the databases and checks if the limits are exceeded:

import common, osproc, os, json, strutils

proc get(file, value, time: string): float =
  let (output, errorCode) = execCmdEx(rrdtool & " graph x -s -" & time & " DEF:v=" & file & ":" & value & ":AVERAGE VDEF:vm=v,AVERAGE PRINT:vm:%lf")

  if errorCode != 0:
    raise newException(ValueError, "Error code " & $errorCode & " from rrdtool: " & output)

  result = output.splitLines[^2].parseFloat

if paramCount() != 1:
  echo "alert [1d|7d|49d]"
  quit 1

let statsJson = parseFile statsJsonFile
for server in statsJson["servers"]:
  let
    domain = server["type"].str
    name = server["name"].str
    fileNet = (rrdDir / domain) & "-net.rrd"
    fileCpu = (rrdDir / domain) & "-cpu.rrd"
    fileMem = (rrdDir / domain) & "-mem.rrd"

  template alert(s: string) = echo name, " ", s

  case paramStr(1)
  of "1d":
    if fileNet.get("network_rx", "4min") + fileNet.get("network_tx", "4min") > 2_000_000:
      alert "network traffic over 2 MB/s for 4 min"

    if fileMem.get("memory_used", "4min") + fileMem.get("swap_used", "4min") > 0.9 * (fileMem.get("memory_total", "4min") + fileMem.get("swap_total", "4min")):
      alert "memory and swap over 90% for 4 min"

  of "7d":
    if fileCpu.get("cpu", "21min") > 90.0:
      alert "CPU over 90% for 21 min"
    if fileCpu.get("load", "21min") > 10.0:
      alert "Load over 10 for 21 min"

  of "49d":
    let network_rx = fileNet.get("network_rx", "4410")
    if network_rx != network_rx: # NaN
      alert "unreachable for 1 hour"

  else:
    echo "unknown parameter ", paramStr(1)
    quit 1

Nim alert footprint:

Part	Runtime	Memory
Alert (9 servers)	0.25 s	2.12 MB

Conclusion

Live view of the last day of the DDNet.tw web server, also hosting this blog:

You can see the full graphs on the DDNet Server Statistics page. As usual you can find the entire source code in our git repository.

All in all the system runs on very little resources, puts out some nice graphs and alerts me automatically about defined problems.

Part	Runtime	CPU	Memory
Client		0.14 %	3.70 MB
Server (9 clients)		0.58 %	0.18 MB
Create (9 servers)	0.01 s		1.74 MB
Graph (9 servers)	1.99 s		2.82 MB
Monitor (9 servers)		0.03 %	0.89 MB
Alert (9 servers)	0.25 s		2.12 MB

Writing an Async Logger in Nim

2016-01-28T00:00:00-05:00

Surprisingly I’m working on HookRace again. I might share a few interesting code snippets and thoughts in this blog along the way. I’m still going with Nim as the programming language.

For an easy start let’s write a logging module that can be used everywhere in the game’s server as well as client. There are mostly three aspects that I care about:

Can be configured to write to different files and/or stdout
Writes to the disk asynchronously, preventing any blocking when the disk is overloaded
Reasonable performance

Follow along if you want to witness how fun it is to write code in Nim!

Motivation

A few logger implementations for Nim already exist:

logging: A simple logger in the standard library. Configurable, but no async writing
omnilog: An advanced logger with lots of features, no async writing
syslog: A bit too high level for my taste and no Windows support

We could adapt the logging or omnilog modules to our requirements, mainly by adding async writing. For now let’s write our own simple logging module from scratch instead. If it turns out to be usable after a few iterations one might merge it with an existing module. Also, this post would be rather boring if I just ended up using an existing logger.

Implementation

Let’s start with the simplest logger possible:

proc log*(text: string) =
  echo text

log "Hello World!"

We have a procedure log that is exported (*). It takes a value text of type string and prints this text to the standard output. Super simple and can be used like this from another file (or the same one):

import logger
log "Hello World!"

This simply prints Hello World to the terminal for now when we execute it. We want to keep using the logger in this exact format.

Next step: Add a fancy logging format:

import strutils, times

proc log*(text: string) =
  echo "[$# $#]: $#" % [getDateStr(), getClockStr(), text]

This uses the strutils and times modules from Nim’s standard library. The % operator formats the string "[$# $#]: $#" with a date, clock and our text into [2016-01-28 20:04:43]: Hello World!

It would also be nice if we could see which module is actually invoking the logger. The instantiationInfo proc can be used to retrieve the filename and line of a template instatiation at compile time. So let’s make our logger a template, this also improves performance because we don’t even need a function call at runtime anymore:

template log*(text: string) =
  const module = instantiationInfo().filename
  echo "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
    module, text]

We end up with this output:

[2016-01-28 20:14:04][example.nim]: Hello World!

Let’s get rid of the .nim at the end:

  const module = instantiationInfo().filename[0 .. ^5]

[0 .. ^5] gives us the part of the string from position 0 (from the start) to position 5 from the end. New output:

[2016-01-28 20:14:04][example]: Hello World!

Perfect! We’ll keep this exact output format.

Now let’s add the ability to log to multiple files simultaneously. We also just handle stdout as a regular file:

type Logger = object
  file: File

var loggers = newSeq[Logger]()

proc addLogger*(file: File) =
  loggers.add Logger(file: file)

template log*(text: string) =
  let
    module = instantiationInfo().filename[0 .. ^5]
    str = "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
      module, text]
  for logger in loggers:
    logger.file.writeLine str
    logger.file.flushFile

By flushing the file we make sure that the output is actually written to the file right now, not just once the buffer is full. That’s important to us, for example if the program crashes and we’d lose the last few lines of the log otherwise.

Now we have to register a few loggers before we can log something, but that’s the only change:

import logger

addLogger stdout
addLogger open("example.log", fmWrite)

log "Hello World!"

Now we write our log message to stdout as well as a file. If we used the fmAppend file mode instead of fmWrite we could append to the log file, right now we’re overwriting it every time we run our example program.

Here comes our main problem: What happens if we’re logging while the game server is running regularly, but the write to the disk takes a few milliseconds or even longer! The game server gets delayed and our performance is terrible!

Adding a Thread

So let’s add a separate thread to write the log messages. In Nim each thread has its own heap, so we can’t directly pass a string to another thread. Instead we will communicate with our new thread using a channel, similarly to Go channels. We start by defining a Message type to send over our channel:

type
  MessageKind = enum
    write, update, stop

  Message = object
    case kind: MessageKind
    of write:
      module, text: string
    of update:
      loggers: seq[Logger]
    of stop:
      nil

var channel: Channel[Message]

We define three kinds of messages:

write: Write a new log message
update: Update the set of files to log to
stop: Stop the logging thread

Our new proc for logging looks like this:

proc threadLog {.thread.} =
  var loggers = newSeq[Logger]()

  while true:
    let msg = recv channel
    case msg.kind
    of write:
      let str = "[$# $#][$#]: $#" % [getDateStr(), getClockStr(),
        msg.module, msg.text]

      for logger in loggers:
        logger.file.writeLine str
        logger.file.flushFile
    of update:
      loggers = msg.loggers
    of stop:
      break

The thread keeps running until it receives a stop message. Calling recv on a channel blocks until a message is received. Now that the main load is moved to a new thread, what do we do in our log template and addLogger proc?

template log*(t: string) =
  const module = instantiationInfo().filename[0 .. ^5]
  channel.send Message(kind: write, module: module, text: t)

proc addLogger*(file: File) =
  loggers.add Logger(file: file)
  channel.send Message(kind: update, loggers: loggers)

Simple enough! We also need to initialize the channel and run the thread:

open channel
var thread: Thread[void]
thread.createThread threadLog

Now we have to compile with nim --threads:on c example to get thread support, or we simply add a nim.cfg that contains threads: on. Finally we run our code again and … nothing happens, no log message is printed.

The problem is that we finish the program too quickly. This kills the thread before it can receive our logging message. So let’s add a proc that manages the shutdown process more gracefully:

proc stopLog {.noconv.} =
  channel.send Message(kind: stop)
  joinThread thread
  close channel

  for logger in loggers:
    if logger.file notin [stdout, stderr]:
      close logger.file

addQuitProc stopLog

That’s it, our logger works! For more convenience we can add log levels and corresponding templates:

type Level* {.pure.} = enum
  debug, info, warn, error, fatal

var logLevel* = Level.info

template debug*(t: string) =
  if logLevel <= Level.debug:
    const module = instantiationInfo().filename[0 .. ^5]
    channel.send Message(kind: write, module: module, text: t)

template info*(t: string) =
  if logLevel <= Level.info:
    const module = instantiationInfo().filename[0 .. ^5]
    channel.send Message(kind: write, module: module, text: t)

...

What if we want to log not just strings? We can instead accept any kind and number of arguments and stringify them automatically using $:

proc send(module: string, args: varargs[string]) =
  var msg = Message(kind: write, module: module, text: "")
  for arg in args:
    msg.text.add arg
  channel.send msg

template log*(args: varargs[string, `$`]) =
  const module = instantiationInfo().filename[0 .. ^5]
  send module, args

Now we can call our logger like this:

var x = 12
info "All ", x, " Bananas"

[2016-01-28 21:57:34][example]: All 12 Bananas

For more advanced formatting one might want to use the excellent strfmt library:

import strfmt
var x = 12
var y = 3.4
info "{} + {:.2f} == {}".fmt(x, y, x.float + y)
info interp"I have $x Bananas!"

[2016-01-28 22:24:01][example]: 12 + 3.40 == 15.4
[2016-01-28 22:24:01][example]: I have 12 Bananas!

Everything works great and we can be sure that hard disk latency will not bite us. But how is the performance?

Optimization

First let’s come up with a simple program that allows us to measure performance:

import logger

addLogger open("bench.log", fmWrite)
for i in 1 .. 1_000_000:
  log "Hello World!"

Build Command	Runtime
`nim c bench` (unoptimized)	31.9 s
`nim -d:release c bench` (optimized)	12.1 s

12 seconds to write a million log messages. I guess that’s good enough for my use case. We’d rather run out of free disk space than have too little performance like this.

Anyway, let’s see if there are any low hanging fruits. We could look at the code and try to isolate some parts to figure out what takes long. Or we compile the benchmark with debugger support using nim -d:release --debugger:native c bench and invoke it with valgrind --tool=callgrind ./bench. Afterwards kcachegrind can help us analyze the function calls:

What stands out is that getDateStr and getClockStr take 33% and 32% respectively of the total time! Since the time string only changes once every second we can cache it:

proc threadLog {.thread.} =
  var
    loggers = newSeq[Logger]()
    lastTime: Time
    timeStr = ""

  while true:
    let msg = recv channel
    case msg.kind
    of write:
      let newTime = getTime()
      if newTime != lastTime:
        timeStr = getLocalTime(newTime).format "yyyy-MM-dd HH:mm:ss"
        lastTime = newTime

      let str = "[$#][$#]: $#" % [timeStr, msg.module, msg.text]

      for logger in loggers:
        logger.file.writeLine str
        logger.file.flushFile
    ...

getTime() gets the time in seconds, so we only format a new timeStr when a new second has arrived.

Code Change	Runtime
Cache time string	4.4 s

Flushing the file after each write still consumes a lot of time. Unfortunately this can’t be seen using valgrind because valgrind’s instrumentation slows down the entire program so much that the flushing itself becomes insignificant.

The trick is to flush the file only when we’re actually done writing, that means when we have no more messages waiting in the channel:

if channel.peek == 0:
  logger.file.flushFile

Code Change	Runtime
Flush only when necessary	2.1 s

A general tip when optimizing Nim code is to prevent string allocations and copying. Instead of using writeLine we can add the newline directly and call write:

let str = "[$#][$#]: $#\n" % [timeStr, msg.module, msg.text]

for logger in loggers:
  logger.file.write str

We can use the same message and simply reset its text string instead of allocating a new one every time:

var msg = Message(kind: write, system: "", text: "")
proc send(moduleName: string, args: varargs[string, `$`]) =
  msg.system = moduleName
  msg.text.setLen(0)
  for arg in args:
    msg.text.add arg
  channel.send msg

These two changes don’t have a huge effect, but it’s better than nothing:

Code Change	Runtime
Prevent allocations	1.8 s

Another idea is to use a non-locking write since we only write from one thread anyway. The POSIX C function fwrite_unlocked can easily be imported and used in Nim:

proc fwriteUnlocked(buf: pointer, size, n: int, f: File): int {.
  importc: "fwrite_unlocked", noDecl.}

proc writeUnlocked(f: File, s: string) =
  if fwriteUnlocked(cstring(s), 1, s.len, f) != s.len:
    raise newException(IOError, "cannot write string to file")

logger.file.writeUnlocked str

Unfortunately this does not have any effect on performance.

So now we can log "Hello World" a million times in 1.8 seconds. For comparison, simply printing "Hello World" a million times, without any formatting or asynchronous writing, takes 2.4 seconds already when redirecting stdout to a file:

for i in 1 .. 1_000_000:
  echo "Hello World!"

This is of course because of the flushing that happens internally in echo. If we opt to use a raw write directly we can write the million strings in just 0.2 s, so there is still headroom for further improvements.

Final words

We implemented a simple logger, saw a few cool Nim features along the way and finally improved the performance by a factor of 6.

An even better optimization would be not to use channels for inter-thread communication. After all even the channels documentation notes that they are slow. Instead we could create a data structue in the heap shared between threads and use this. But I doubt that I will need more performance for logging, so this is good enough for now.

The resulting code of this post and future HookRace code can be found on GitHub.

Nim binary size from 160 KB to 150 Bytes

2015-05-04T00:00:00-04:00

The size of binaries in the Nim programming language seems to be a popular topic recently. Nim’s slogan is expressive, efficient, elegant, so let’s examine the efficient part in this post by exploring a few ways to reduce the size of a simple Nim Hello World binary on Linux. Along the way we will:

Build a regular program into a 6 KB binary
Disregard the C standard library and build a 952 byte binary
Use a custom linker script and ELF header to build a 150 byte binary
(1 byte smaller than in Rust)

The full source code of this post can be found in its repository. All measurements are done on Linux x86-64 with GCC 5.1 and Clang 3.6.0.

Using the C Standard Library

echo "Hello!"

By default Nim uses GCC as the backend C compiler on most platforms, and we dynamically link against glibc. We can try optimizing for speed and size, as well as strip unnecessary symbols after the compilation:

Command (GCC backend)	Binary Size
`nim c hello`	160 KB
`nim -d:release c hello`	61 KB
`nim -d:release --opt:size c hello`	25 KB
`nim -d:release --opt:size c hello && strip -s hello`	19 KB

That’s pretty nice and can be done with any Nim program to reduce binary size.

Now let’s try to get rid of glibc, at least temporarily (we will come back to a more permanent solution later). Instead of glibc we’re now statically linking against musl libc:

$ nim --gcc.exe:/usr/local/musl/bin/musl-gcc \
  --gcc.linkerexe:/usr/local/musl/bin/musl-gcc \
  -d:release --opt:size --passL:-static c hello
$ strip -s hello
30 KB

Update: The order of arguments to nim matters, --passL:-static has to be passed after setting the gcc exe so that it isn’t overwritten.

So that’s a statically linked binary in 30 KB, which can be deployed without depending on any glibc version (or any other libraries)!

What about using Clang instead of GCC by setting --cc:clang:

Command (Clang backend)	Binary Size
`nim --cc:clang c hello`	168 KB
`nim --cc:clang -d:release c hello`	33 KB
`nim --cc:clang -d:release --opt:size c hello`	29 KB
`nim --cc:clang -d:release --opt:size c hello && strip -s hello`	23 KB

The speed optimized binary is much smaller, but the one optimized for size isn’t. The exact behaviour of Clang and GCC of course depends on their version, so expect to see (at least) slightly different numbers on your system.

Looks like the GCC backend is a better choice currently, so let’s try stripping down the binary further with it:

As a first step we disable the garbage collector, not like we need it for this program anyway:

$ nim --gc:none -d:release --opt:size c hello
$ strip -s hello
11 KB

Next we remove all the nice dynamic memory, error handling and other OS dependent goodies with --os:standalone (this implies --gc:none). We have to provide a panicoverride.nim that contains these two procs, about which we don’t care anyway. Who needs error handling when you can have a 6 KB binary instead:

proc rawoutput(s: string) = discard
proc panic(s: string) = discard

$ nim --os:standalone -d:release c hello
$ strip -s hello
6.1 KB

Disregarding the C standard library

Now we have to start thinking more out of the box: If we want a program that really does nothing, not even print Hello!, we can just use an empty file. Now we don’t rely on the C standard library anymore and can try to exclude it totally by using -passL:-nostdlib (passL simply passes that argument to GCC at the linking step):

$ nim --os:standalone -d:release --passL:-nostdlib c hello
CC: hello
CC: stdlib_system
[Linking]
ld: warning: cannot find entry symbol _start; defaulting to 0000000000400160
$ strip -s hello
1.4 KB

Wow, that’s small! Let’s run our program, that does nothing, and enjoy:

$ ./hello
Segmentation fault (core dumped)

Ouch. That didn’t go so well. Take another look at the linker output and notice what the problem is: We can’t just expect our code to be run, the binary starts its execution at some random, wrong, position. Instead we have to take over the work of the C standard library now and supply our own _start function:

import syscall

proc main {.exportc: "_start".} =
  discard syscall(EXIT, 0)

We also have to explicitly exit the program, for which we use my syscall library, that provides raw system calls into the Linux kernel in Nim. Let’s wrap the syscalls we need and use them to write proper Nim code with them:

import syscall

const STDOUT = 1

proc write(fd: cint, buf: cstring, len: csize): clong
          {.inline, discardable.} =
  syscall(WRITE, fd, buf, len)

proc exit(n: clong): clong {.inline, discardable.} =
  syscall(EXIT, n)

proc main {.exportc: "_start".} =
  write STDOUT, "Hello!\n", 7
  exit 0

Now we can successfully compile like this:

$ nim --os:standalone -d:release --passL:-nostdlib --noMain c hello
$ strip -s hello
1.5 KB
$ ./hello
Hello!

The final trick of this section is to tell GCC to optimize away unused functions. These functions are the initializations of Nim modules, like our hello module or the system module from the standard library, but they just end up empty here anyway. Maybe the Nim compiler could leave them out on its own, but usually you don’t care about saving a few bytes and have something more important to do. Today we care, so first we tell GCC to put function and data items into separate sections (-ffunction-sections & -fdata-sections) at the compile step. At the linking step we tell Nim to tell GCC to pass --gc-sections to ld, our linker, which then removes sections that are not referenced anywhere:

$ nim --os:standalone -d:release --passL:-nostdlib --noMain \
  --passC:-ffunction-sections --passC:-fdata-sections \
  --passL:-Wl,--gc-sections c hello
$ strip -s hello
952 B

Great! We’re down from an initial binary size of 160 KB to just 952 bytes. Can we get even smaller? Yes, indeed, but not with the default tooling.

Custom Linking to achieve 150 Bytes

This uses the exact method from the 151-byte static Linux binary in Rust blog post, except that Nim with GCC manage to get down 1 byte more. Meanwhile Clang needs 1 byte more than the Rust version.

We continue with the exact same program that we just got down to 952 bytes. But instead of letting Nim do all the work (with quite a lot of options by now) we simply create an object file from Nim (--app:staticlib), and go manually from there. A custom linker script and a custom ELF header do the main work. But the actual code that’s executed is still fully provided by our Nim code:

$ nim --app:staticlib --os:standalone -d:release --noMain \
  --passC:-ffunction-sections --passC:-fdata-sections \
  --passL:-Wl,--gc-sections c hello
$ ld --gc-sections -e _start -T script.ld -o payload hello.o
$ objcopy -j combined -O binary payload payload.bin
$ ENTRY=$(nm -f posix payload | grep '^_start ' | awk '{print $3}')
$ nasm -f bin -o hello -D entry=0x$ENTRY elf.s
$ chmod +x hello
$ wc -c < hello
158
$ ./hello
Hello!

158 bytes! There’s a final trick to get 8 bytes smaller. We put our string inside the padding in the ELF header and access that memory manually:

proc main {.exportc: "_start".} =
  write STDOUT, cast[cstring](0x00400008), 7
  exit 0

$ wc -c < hello
150
$ ./hello
Hello!

150 bytes! That’s the final size we’re going to get. If you still don’t have enough and want to write your binary manually you can use even more hacks to get down to 45 bytes as described in the excellent Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux.

Summary

Nim is fine for writing small binaries. Now you also know how to write Nim without the C standard library. It might be fun to write your own runtime in Nim from scratch. You can check out the repository to get your own results:

$ ./run.sh
== Using the C Standard Library ==
hello_unoptimized    163827
hello_release         62131
hello_optsize         25248
hello_optsize_strip   18552
hello_gcnone          10344
hello_standalone       6208

== Disregarding the C Standard Library ==
hello2                 1776
hello3                  952

== Custom Linking ==
hello3_custom           158
hello4_custom           150

$ objdump -rd nimcache/hello4.o
...
0000000000000000 <_start>:
 0: b8 01 00 00 00          mov    $0x1,%eax
 5: ba 07 00 00 00          mov    $0x7,%edx
 a: be 08 00 40 00          mov    $0x400008,%esi
 f: 48 89 c7                mov    %rax,%rdi
12: 0f 05                   syscall 
14: 31 ff                   xor    %edi,%edi
16: b8 3c 00 00 00          mov    $0x3c,%eax
1b: 0f 05                   syscall 
1d: c3                      retq 
...

Discussions on Hacker News and Reddit.

Addendum

I did this for 32bit x86 now as well, which results in a 119 byte binary with GCC and 118 byte with Clang, more information in the repository.

With a simple patch to the Nim compiler the {.noReturn.} pragma now actually removes the final retq call that is useless after the EXIT syscall. So the final binary size now is 149 bytes with x86-64, 116 bytes with x86.

Porting a NES emulator from Go to Nim

2015-05-01T00:00:00-04:00

Let me get this straight. We have an emulator for 1985 hardware that was written in a pretty new language (Go), ported to a language that isn’t even 1.0 (Nim), compiled to C, then compiled to JavaScript? And the damn thing actually works? That’s kind of amazing.

— Summary by haberman

I spent the last weeks working on NimES, a NES emulator in the Nim programming language. As I really liked fogleman’s NES emulator in Go I ended up mostly porting it to Nim. The source code is so clean that it’s often easier to understand the internals of the NES by reading the source code than by reading documentation about it.

The choice of backend fell on SDL2 for me, contrary to GLFW + PortAudio that the Go version used. This was mainly motivated by the great portability promised by SDL2. Later we will see how porting to JavaScript and Android worked. If you’re impatient and want to play a game, there’s a JS demo.

Comparison of Go and Nim

Most Go concepts are quite trivial to translate to Nim. This made the porting process simple.

Let’s compare some data that I found interesting:

Metric	Go	Nim (clang backend)
Compile command	`go build`	`nimble build`
Fresh compile time	2.1 s¹	1.7 s
Recompile time	1.5 s	0.5 s
Binary size	16 MB (static)	136 KB + 1MB SDL2
Source code size²	3260 lines, 74 KB	2145 lines, 60 KB
CPU usage	71 %	53 %
Memory usage	79 MB	73 MB

¹ Excluding go-glfw, go-gl and portaudio, which take 17 s to compile
² Emulation code only

It’s nice to see Nim doing well. Even the compile time is shorter than that of Go, which is well known for its short compile times. Now that the port seems to be doing fine and should be running on all Desktop platforms, let’s look into some other interesting things we can do with Nim:

JavaScript port via emscripten

Nim has a JavaScript backend, but I don’t trust it to be stable enough for this task yet. So I opted for emscripten instead, which can compile C code into JavaScript. Since Nim outputs C code, this sounds like a perfect fit. Luckily eeeee helped me with getting it started, since he had experience by porting my DDNet client to teewebs.net.

It turned out that emsdk is the easiest way to use emscripten:

$ ./emsdk update
$ ./emsdk install latest
$ ./emsdk activate latest
$ source ./emsdk_env.sh

This may take a while, get a cup of tea. Afterwards we should have the emconfigure, emmake and emcc commands available. We can build regular Nim programs and look at the resulting html file:

$ cat hello.nim
echo "Hello World"
$ nim --cc:clang --clang.exe:emcc --clang.linkerexe:emcc \
  --cpu:i386 -d:release -o:hello.html c hello.nim
$ ls -lha hello.{html,js}
-rw-r--r-- 1 def users 101K Mai  1 19:02 hello.html
-rw-r--r-- 1 def users 385K Mai  1 19:02 hello.js

That’s a pretty cumbersome building command, so we’ll slim it down later. The next step is to build SDL2 for emscripten:

$ hg clone https://hg.libsdl.org/SDL
$ cd SDL
$ emconfigure ./configure --host=asmjs-unknown-emscripten \
  --disable-assembly --disable-threads \
  --enable-cpuinfo=false CFLAGS="-O2"
$ emmake make
$ ls -lha build/.libs/libSDL2.a
-rw-r--r-- 1 def users 1.6M Apr 29 06:58 build/.libs/libSDL2.a

I put the resulting libSDL2.a into the NimES repository under emscripten/ for convenience.

Instead of increasing the cumbersomeness of our build command anymore, NimES’s nim.cfg specifies how to compile when -d:emscripten is set:

@if emscripten:
  define = SDL_Static
  gc = none
  cc = clang
  clang.exe = "emcc"
  clang.linkerexe = "emcc"
  clang.options.linker = ""
  cpu = "i386"
  out = "nimes.html"
  warning[GcMem] = off
  passC = "-Wno-warn-absolute-paths -Iemscripten -s USE_SDL=2"
  passL = "-O3 -Lemscripten -s USE_SDL=2 --preload-file tetris.nes --preload-file pacman.nes --preload-file smb.nes --preload-file smb3.nes -s TOTAL_MEMORY=16777216"
@end

Now a simple nim -d:release -d:emscripten c src/nimes builds the JavaScript port. Note that I’m preloading a few ROMs so that they can be loaded. The HTML then uses the ?nes= parameter to pass the command line argument:

var argument;
if (QueryString.hasOwnProperty("nes")) {
  argument = QueryString.nes;
} else {
  argument = "smb3.nes";
}

var Module;
Module = {
  preRun: [],
  postRun: [],
  arguments: [argument],
  canvas: (function() {
    var canvas = document.getElementById('canvas');
    canvas.addEventListener("webglcontextlost", function(e) { alert('WebGL context lost. You will need to reload the page.'); e.preventDefault(); }, false);
    return canvas;
  })(),
  totalDependencies: 0
};

Inside the Nim source code there are some interesting changes too. I quickly wrapped these functions as there is no emscripten wrapper for Nim yet:

when defined(emscripten):
  proc emscripten_set_main_loop(fun: proc() {.cdecl.}, fps,
    simulate_infinite_loop: cint) {.header: "<emscripten.h>".}

  proc emscripten_cancel_main_loop() {.header: "<emscripten.h>".}

Emscripten requires a slightly different execution style. Instead of actually looping, we define the main loop like this:

when defined(emscripten):
  emscripten_set_main_loop(loop, 0, 1)
else:
  while runGame:
    loop()

That’s the main idea and with this we get a pretty playable web version of NimES. I’m still getting 60fps in it, but just barely on my machine. Chrome seems to do a bit better than Firefox.

Android port

Obviously the next step is to port NimES to Android as well. But since the original emulator is more accurate and nice than performant, we shouldn’t expect runnable speed. Think of this more as a proof of concept:

We need a fresh clone of the SDL2 repository for this as well as the Android SDK (12 or later) and NDK (7 or later) installed. SDL2 has building instructions for Android as well:

$ hg clone https://hg.libsdl.org/SDL
$ cd SDL/build-scripts
$ ./androidbuild.sh org.nimes /dev/null
$ ls ../build/org.nimes
gen/  src/                 build.properties    local.properties
jni/  AndroidManifest.xml  build.xml           proguard-project.txt
res/  ant.properties       default.properties  project.properties

That’s our Android build directory now. I put this into the repository as well, under android/. Now we can add some ROM to the assets/ directory and tell Nim to put the resulting C files into the correct directory and not to build them into binaries at all:

@if android:
  cpu = "i386"
  nimcache = "./android/jni/src"
  compileOnly
  noMain
@end

You may have noticed that I also defined noMain. Instead we define our own main function, as SDL is a bit weird with mains. Thanks to yglukhov for this little trick:

when defined(android):
  {.emit: """
  #include <SDL_main.h>

  extern int cmdCount;
  extern char** cmdLine;
  extern char** gEnv;

  N_CDECL(void, NimMain)(void);

  int main(int argc, char** args) {
      cmdLine = args;
      cmdCount = argc;
      gEnv = NULL;
      NimMain();
      return nim_program_result;
  }

  """.}

Another trick is how to access the assets we embed into our APK. Luckily SDL2 provides functions for that, which we can use as replacements for the regular file operations:

from sdl2 import rwFromFile, read, freeRW

proc newCartridge*(path: string): Cartridge =
  var file = rwFromFile(path.cstring, "r")
  defer: freeRW file

  var header: iNESHeader
  # Read directly into the header object
  if read(file, addr header, 1, sizeof header) != sizeof header:
    raise newException(ValueError, "header can't be read")
  ...

Finally we can build the project:

$ nim -d:release -d:android c src/nimes
$ cd android
$ ndk-build
$ ant debug

And the end result is a nice nimes.apk. Of course it only shows some low FPS video for now and doesn’t even have any controls, but it’s a start.

Conclusion

In the end I’m quite happy with the result: A truly portable emulator written in my favorite language. It compiles to C, C++ as well as JavaScript and runs on any Desktop platform as well as JavaScript and Android. The process for this was much easier than expected, mostly thanks to Nim and SDL2. I see a bright future for Nim as a practical language.

If you have any comments, suggestions or questions, feel free to ask them on Hacker News or Reddit.

NimES: a NES emulator in Nim

2015-04-30T00:00:00-04:00

Since today Nim 0.11 has been released, I guess it’s a good time to release this as well: NimES is a NES emulator written in Nim. More info in the GitHub repository and the JS demo.

Discussion on Hacker News and Reddit.

Make a Lisp in Nim

2015-03-04T00:00:00-05:00

I spent the last weekend working through the amazing guide for Make a Lisp, writing a Lisp interpreter in Nim. The final result just made it into the repository.

Running the Nim version is pretty simple. You need the Nim compiler from the devel branch and nre which can be installed through nimble. After installing those you can build and run the MAL interpreter:

$ cd nim
$ make
  # OR
$ nimble build
$ ./stepA_mal
Mal [nim]
user> 12
12
user> (+ 2 3)
5

Running the tests:

$ cd ..
$ make "test^nim^step1" # A single test
$ make "test^nim" # All tests
$ make "perf^nim" # All benchmarks

There is a nice presentation in MAL about MAL you can read:

$ cd nim
$ ./stepA_mal ../examples/clojurewest2014.mal

And you can run MAL implented in MAL itself using the Nim interpreter:

$ ./stepA_mal ../mal/stepA_mal.mal
Mal [nim-mal]
mal-user>

The benchmark results from Joel Martin, the author of MAL, don’t look bad for Nim. Edit: Note that these are just rough measurements to see that the Nim implementation is doing fine. Don’t judge the other languages for their numbers, which may not have ideal implementations performance-wise.

In the short benchmarks Nim is the fastest, in the long benchmark only the JVM can beat it. I didn’t really try to optimize and oriented mostly on the Python implementation, which is quite a lot slower as you can see. So it’s pretty nice to see idiomatic Nim performing well:

             perf1  perf2       perf3
            macros   math      macros
                ms     ms   iters/sec
java             6     24       17969
scala           47     87       15963
nim              1      1       11121
ocaml            1      3        7063
cs              10     11        5414
vb              12     13        4523
rust             2      5        4084
c                1      4        3649
go               1      6        3048
racket           3     10        2461
coffee           5     11        2326
js               6     14        1726
ruby             4     15        1255
haskell          4     14        1163
clojure         11     23        1174
forth            9     29         563
php             13     51         331
python          14     51         304
bash          1673   9000         276
perl            18     69         215
ps              41    332          48
R              116    490          28
miniMAL        779   3448           4
matlab        1688   5844           2
make          3427  28453           0

I also didn’t really aim for a low amount of code but rather good readability, but it may still be interesting to look at. I guess Nim’s size being so close to Python shows how much I oriented on the Python implementation:

                Lines   Words   Chars
mal               280     913    7075
clojure           369    1164   10099
ruby              481    1513   13247
coffee            532    2020   15653
racket            550    1937   17229
python            670    1997   19632
nim               715    2453   20263
r                 812    2283   21644
lua               914    2640   23102
ocaml             597    3704   24371
scala             871    3060   24885
js                920    3059   26437
php               940    3011   27332
matlab            959    2485   28322
perl             1081    3511   29091
miniMAL           857    3104   29983
haskell           985    4452   30115
bash             1347    3586   31717
go               1297    4744   36321
ps               1409    6117   38651
forth            1609    6826   44715
cs               1249    4136   45039
java             1591    4966   53223
rust             1897    5702   56516
vb               1556    5175   58099
make             1593    6055   62310
c                2304    6957   73047

Thanks to Joel Martin for this guide. This was my first time getting close to Lisp and I recommend others to work through the MAL guide as well. But be aware, once you get through the first steps, it becomes addictive.

Comments on Reddit and Hacker News.

How I Start: Nim

2015-02-09T00:00:00-05:00

My most recent post goes in another direction (no, it’s still about Nim): How I start a new Nim project. Check out the article on howistart.org. Comments on Hacker News and Reddit as always.

Conclusion on Nim

2015-01-26T00:00:00-05:00

In my last two posts, “What is special about Nim?” and “What makes Nim practical”, I forgot the important conclusion - why I personally have decided for Nim in favor of Rust, C++, Python and Haskell:

Nim is not the fastest language, it’s not the easiest language to write in and it surely has some flaws that should be fixed. Nim has no single “killer feature” like go’s goroutines or Rust’s memory management. But Nim doesn’t need a killer feature. Instead it strikes a reasonable balance that makes it the most efficient language for me:

I can produce reasonably efficient code (faster than Python and Haskell)
that is reasonably readable (more so than Rust, C++ and Haskell)
in a reasonable amount of time (less than Rust, C++ and Haskell)

What makes Nim practical?

2015-01-23T00:00:00-05:00

In my last post I showed what makes the Nim programming language special. Today, let’s consider Nim from another angle: What makes Nim a practical programming language?

Binary Distribution

Programs written in interpreted languages like Python are difficult to distribute. Either you require Python (in a specific version even) to be installed already, or you ship it with your program. This even causes some to reconsider Python as a teaching language.

How does Nim work around this problem? For starters your program gets statically linked against the Nim runtime. That means you end up with a single binary that depends solely on the standard C library, which we can take for granted on any operating system we’re interested in.

Let’s write a small program and give this a try:

echo "Hello World"

If you want to follow along, get the Nim compiler. Save this code as hello.nim. Let’s compile it now so that we can distribute the hello binary:

$ nim -d:release c hello
CC: hello
CC: stdlib_system
[Linking]
$ ./hello
Hello World
$ ls -lha hello
-rwxr-xr-x 1 def def 57K Jan 22 09:37 hello*
$ ldd hello
    linux-vdso.so.1 (0x00007fffd5973000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f0f92c6b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0f928c3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0f92e6f000)

Now our little program starts growing and we get interested in using a few Nim libraries. Do we have to add compiled versions of these libraries to our distributions now? Do not fret! Nim libraries are statically compiled into our binary as well. Let’s get a library using Nim’s package manager, nimble:

$ nimble install strfmt
Installing strfmt-0.5.4
strfmt installed successfully.

And use the library in our, admittedly not very useful, program:

import strfmt

echo "Hello {} number {:04.1f}".fmt("World", 6.0)

If we want to compile with Clang instead of GCC as the backend, that’s easy as well and Clang is usually much faster to compile and yields a smaller binary:

$ nim --cc:clang -d:release c hello
$ ./hello
Hello World number 06.0
$ ls -lha hello
-rwxr-xr-x 1 def def 54K Jan 22 10:06 hello*

The only dynamic libraries we have to worry about are the C libraries we use, but that’s a story for another time.

Source Distribution

We can’t compile our program for all possible combinations of operating systems and CPU architectures of course. But we can be prepared for them and create a source distribution that can be compiled on the target systems with just a C compiler:

$ tree
.
├── hello.ini
├── hello.nim
└── lib
    └── nimbase.h
$ cat hello.ini
[Project]
Platforms: """
  windows: i386;amd64
  linux: i386;amd64;powerpc64;arm;sparc;mips;powerpc
  macosx: i386;amd64;powerpc64
  solaris: i386;amd64;sparc
  freebsd: i386;amd64
  netbsd: i386;amd64
  openbsd: i386;amd64
  haiku: i386;amd64
"""

[Lib]
Files: "lib/nimbase.h"

[Windows]
binPath: "bin"
$ niminst csource hello.ini -d:release

Here’s the resulting build script if you want to try it out on any of these platforms: hello-csources.zip

$ unzip hello-csources.zip
$ ./build.sh
SUCCESS
$ bin/hello
Hello World

niminst can also be used to build full installers for your program. It’s what the Nim compiler itself uses.

Debug and Release Builds

Debug and release builds behave quite differently in Nim by default. Here’s an overview:

Option	Debug build	Release build
objChecks	On	Off
fieldChecks	On	Off
rangeChecks	On	Off
boundChecks	On	Off
overflowChecks	On	Off
assertions	On	Off
stackTrace	On	Off
lineTrace	On	Off
lineDir	On	Off
deadCodeElim	Off	On

As you develop your program you will be thankful for the range, bound and overflow checks. As you release your program your users will be thankful for the additional speed of disabling those.

If this is not what you want, and you require runtime checks even in release builds, you can enable them project wide or for parts of your code.

For example here we have an expensive max function that takes most of the running time:

proc max[T](x: varargs[T]): T =
  if x.len == 0:
    return

  result = x[0]

  for i in 1 .. x.high:
    if result < x[i]:
      result = x[i]

var s = newSeq[int]()
for i in 1..1_000:
  s.add(i)

echo max(s)

We want our program to never access out of sequence bounds, so we want to compile with nim -d:release --checks:on max. As it’s unreasonable to write this every time, we can create a max.nim.cfg file in our directory instead and enable runtime checks for every compilation of our program:

checks: on

Now we run into the problem that the max function runs too slowly (it doesn’t, but let’s imagine). So we disable runtime checks just for this function:

{.push checks: off.}
proc max[T](x: varargs[T]): T =
  if x.len == 0:
    return

  result = x[0]

  for i in 1 .. x.high:
    if result < x[i]:
      result = x[i]
{.pop.}

Great, now we can control which part of our program has runtime checks and which doesn’t. checks enables and disables all runtime checks at once, but there are more fine-grained controls as well.

Nim instead of Python + C++

High performance languages like C++ may require some boilerplate. A higher level language can be used at compile time to automatically create the boilerplate. In DDNet (which inherited them from Teeworlds) there are many examples for this.

A really simple use case is to get the current git revision at compile time in Python and put it into the config.h with a #define so it can be referred to at runtime in the program. In Nim you need neither Python nor a C preprocessor and can instead do it all directly at compile time in Nim:

const gitHash = staticExec "git rev-parse HEAD"
echo "Revision: ", gitHash

Use as a scripting language

With TinyCC as the backend Nim makes for a nice scripting language. All we need is a nimscript file and the TinyCC compiler:

#!/bin/sh
nim --cc:tcc --verbosity:0 -d:release -r c $*

To automatically update DDNet’s map files on our upcoming HTTP map server I’m using this script (with this crc32 module):

#!/usr/bin/env nimscript

import os, crc32, strutils

const
  baseDir = "/home/teeworlds/servers"
  mapdlDir = "/var/www-maps"

for kind, path in walkDir baseDir/"maps":
  if kind != pcFile:
    continue

  let (dir, name, ext) = splitFile(path)

  if ext != ".map":
    continue

  let
    sum = crc32FromFile(path).int64.toHex(8).toLower
    newName = name & "_" & sum & ext
    newPath = mapdlDir / newName
    tmpPath = newPath & ".tmp"

  if existsFile newPath:
    continue

  copyFile path, tmpPath
  moveFile tmpPath, newPath

Of course you can always compile your program if you need the full speed of Clang or GCC. But for quick tests while developing, or scripts you just need occasionally, this is good enough.

There is a nicer version of this idea now, called nimrun

Debugging Nim

GDB works rather well with Nim. The only problem I’ve had so far is that the variables get unique identifiers in C, but using tab completion you can get along fine:

echo "Hello World"
var x = 10
echo "Value of x: ", x

$ nim --lineDir:on --debuginfo c hello
$ gdb ./hello
(gdb) break hello.nim:3
Breakpoint 1 at 0x41f886: file /home/def/hello.nim, line 3.
(gdb) run
Starting program: /home/def/hello
Hello World
Breakpoint 1, helloInit () at /home/def/hello.nim:3
3       echo "Value of x: ", x
(gdb) print x_89010
$1 = 10
(gdb) print x_89010 = 200
$2 = 200
(gdb) c
Continuing.
Value of x: 200

Wrapping libraries with c2nim

Nice C libraries can automatically be converted into a Nim wrapper with c2nim. Let’s do this for Bellard’s new BPG image format. Since the library is so new, there is no shared library compilation included yet, so I added that myself in my fork.

We need to fix up the C header for c2nim by adding this at the top of libbpg.h:

#ifdef C2NIM
#  dynlib bpglib
#  cdecl

/* Dynamically link to the correct library for our system: */
#  if defined(windows)
#    define bpglib "libbpg.dll"
#  elif defined(macosx)
#    define bpglib "libbpg.dylib"
#  else
#    define bpglib "libbpg.so"
#  endif

/* Remove prefixes in our wrapper, we have modules in Nim: */
#  prefix bpg_decoder_
#  prefix BPG_
#  prefix BPG

/* These are not recognized by c2nim, but that's easy to fix: */
#  mangle uint8_t uint8
#  mangle uint16_t uint16
#  mangle uint32_t uint32
#endif

I made a patch so I can apply the changes again when the libbpg interface changes. Creating the wrapper is now as simple as this:

$ patch -p1 < libbpg.h.patch
patching file libbpg.h
$ c2nim -o:bpg.nim libbpg.h
Hint: operation successful (155 lines compiled; 0 sec total; 516.528KB) [SuccessX]

Look, no hands! The resulting wrapper doesn’t even look bad. Now we can write a simple program to decode a BPG file and save it in the PPM format:

import bpg, os

proc writePPM(img, filename) =
  var imgInfo: ImageInfo
  discard img.getInfo(addr imgInfo)

  let (w,h) = (imgInfo.width.int, imgInfo.height.int)
  var rgbLine = newSeq[uint8](w * 3)

  var f = open(filename, fmWrite)
  f.writeln "P6\n", w, " ", h, "\n255"

  discard img.start(OUTPUT_FORMAT_RGB24)
  for y in 1..h:
    discard img.getLine(addr rgbLine[0])
    discard f.writeBuffer(addr rgbLine[0], w * 3)

  f.close()

if paramCount() != 1:
  stderr.writeln "Usage: decode img.bpg"
  quit 1

var
  buf = readFile paramStr(1)
  img = bpg.open()

if img.decode(cast[ptr uint8](addr buf[0]), buf.len.cint) < 0:
  stderr.writeln "Could not decode image"
  quit 2

img.writePPM("out.ppm")
img.close()

Note that you can tell from the discard statements where I chose to ignore errors. If you actually want to run this, check out the instructions in the repository.

Final words

That’s Nim from a more practical angle. Hopefully you’ll consider Nim for your next project, there are many libraries available already. Also, the community always needs more helping hands!

To do my part in making Nim more practical, I’m trying to implement a new, working REPL using TinyCC. We may also soon get a working compiler as a service for proper IDE integration, directly from Andreas Rumpf.

For comments use Reddit, Hacker News or get in touch with the Nim community directly. You can reach me personally at dennis@felsing.org.

Thanks to Andreas Rumpf and Dominik Picheta again, for proof-reading this post as well.

What is special about Nim?

2015-01-01T00:00:00-05:00

Russian Translation by frol, Chinese Translation by JiyinYiyong, Japanese Translation by Mutsuha Asada

The Nim programming language is exciting. While the official tutorial is great, it slowly introduces you to the language. Instead I want to quickly show what you can do with Nim that would be more difficult or impossible in other languages.

I discovered Nim in my quest to find the right tools to write a game, HookRace, the successor of my current DDNet game/mod of Teeworlds. Since I’m busy with other projects for now, this blog is now officially about Nim instead, until I find time to continue developing the game.

Easy to get running

Ok, this part is not exciting yet, but I invite you to follow along with the post:

for i in 0..10:
  echo "Hello World"[0..i]

If you want to do so, get the Nim compiler. Save this code as hello.nim, compile it with nim c hello and finally run the binary with ./hello. To immediately compile and run, use nim -r c hello. To use an optimized release build instead of a debug build use nim -d:release c hello. With all of these settings you will see the following output:

H
He
Hel
Hell
Hello
Hello 
Hello W
Hello Wo
Hello Wor
Hello Worl
Hello World

Run regular code at compile time

To implement an efficient CRC32 procedure you need a lookup table. You could compute it at runtime or write it into your code as a magic array. Clearly we don’t want any magic numbers in our code, so we’ll do it at runtime (for now):

import unsigned, strutils

type CRC32* = uint32
const initCRC32* = CRC32(-1)

proc createCRCTable(): array[256, CRC32] =
  for i in 0..255:
    var rem = CRC32(i)
    for j in 0..7:
      if (rem and 1) > 0: rem = (rem shr 1) xor CRC32(0xedb88320)
      else: rem = rem shr 1
    result[i] = rem

# Table created at runtime
var crc32table = createCRCTable()

proc crc32(s): CRC32 =
  result = initCRC32
  for c in s:
    result = (result shr 8) xor crc32table[(result and 0xff) xor ord(c)]
  result = not result

# String conversion proc $, automatically called by echo
proc `$`(c: CRC32): string = int64(c).toHex(8)

echo crc32("The quick brown fox jumps over the lazy dog")

Great, this works and we get 414FA339 as the output. But it would be even better if we could compute the CRC table at compile time. This is as easy as it gets in Nim, instead of the current crc32table creation we use:

# Table created at compile time
const crc32table = createCRCTable()

Yes, that’s right: All we had to do was replace the var with a const. Beautiful, isn’t it? We can write the exact same code and have it run at runtime and compile time. No template metaprogramming necessary.

Extend the language

Templates and macros can be used to get rid of boilerplate, by transforming your code at compile time.

Templates just replace their invocations with their code at compile-time. We can define our own loops like this:

template times(x: expr, y: stmt): stmt =
  for i in 1..x:
    y

10.times:
  echo "Hello World"

So the compiler transforms the times-loop to this regular for-loop:

for i in 1..10:
  echo "Hello World"

If you’re curious about the 10.times: syntax, it’s just a regular call to times with 10 as the first parameter and the following indented block as the second parameter. Instead you could also write times(10):, see Unified Call Syntax.

Or initialize sequences (variable sized arrays) more comfortably:

template newSeqWith(len: int, init: expr): expr =
  var result = newSeq[type(init)](len)
  for i in 0 .. <len:
    result[i] = init
  result

# Create a 2-dimensional sequence of size 20,10
var seq2D = newSeqWith(20, newSeq[bool](10))

import math
randomize()
# Create a sequence of 20 random integers smaller than 10
var seqRand = newSeqWith(20, random(10))
echo seqRand

Macros go a step further and allow you to analyze and manipulate the AST. There are no list comprehensions in Nim, for example, but it’s possible to add them to the language by using a macro. Now instead of this:

var res: seq[int] = @[]
for x in 1..10:
  if x mod 2 == 0:
    res.add(x)
echo res

const n = 20
var result: seq[tuple[a,b,c: int]] = @[]
for x in 1..n:
  for y in x..n:
    for z in y..n:
      if x*x + y*y == z*z:
        result.add((x,y,z))
echo result

You can use the future module and write:

import future
echo lc[x | (x <- 1..10, x mod 2 == 0), int]
const n = 20
echo lc[(x,y,z) | (x <- 1..n, y <- x..n, z <- y..n,
                   x*x + y*y == z*z), tuple[a,b,c: int]]

Add your own optimizations to the compiler

Instead of optimizing your code, wouldn’t you prefer to make the compiler smarter? In Nim you can!

var x: int
for i in 1..1_000_000_000:
  x += 2 * i
echo x

This (pretty useless) code can be sped up by teaching the compiler two optimizations:

template optMul{`*`(a,2)}(a: int): int =
  let x = a
  x + x

template canonMul{`*`(a,b)}(a: int{lit}, b: int): int =
  b * a

In the first term rewriting template we specify that a * 2 can be replaced by a + a. In the second one we specify that ints in a multiplication can be swapped if the first is an integer literal, so that we can potentially apply the first template.

More complicated patterns can also be implemented, for example to optimize boolean logic:

template optLog1{a and a}(a): auto = a
template optLog2{a and (b or (not b))}(a,b): auto = a
template optLog3{a and not a}(a: int): auto = 0

var
  x = 12
  s = x and x
  # Hint: optLog1(x) --> ’x’ [Pattern]

  r = (x and x) and ((s or s) or (not (s or s)))
  # Hint: optLog2(x and x, s or s) --> ’x and x’ [Pattern]
  # Hint: optLog1(x) --> ’x’ [Pattern]

  q = (s and not x) and not (s and not x)
  # Hint: optLog3(s and not x) --> ’0’ [Pattern]

s gets optimized to x directly, r gets optimized to x in 2 successive pattern applications and q ends up as 0 immediately.

If you want to see how term rewriting templates can be used to avoid allocations with bigints, look for the templates starting with opt in the bigints library:

import bigints

var i = 0.initBigInt
while true:
  i += 1
  echo i

Bind to your favorite C functions and libraries

Since Nim compiles down to C, foreign function interfaces are fun.

You can easily use your favorite functions from the C standard library:

proc printf(formatstr: cstring)
  {.header: "<stdio.h>", varargs.}
printf("%s %d\n", "foo", 5)

Or use your own code written in C:

void hi(char* name) {
  printf("awesome %s\n", name);
}

{.compile: "hi.c".}
proc hi*(name: cstring) {.importc.}
hi "from Nim"

Or any library you please with the help of c2nim:

proc set_default_dpi*(dpi: cdouble) {.cdecl,
  importc: "rsvg_set_default_dpi",
  dynlib: "librsvg-2.so".}

Control the garbage collector

To achieve soft realtime, you can tell the garbage collector when and for how long it is allowed to run. The main game logic can be implemented like this in Nim, to prevent the garbage collector from causing stutters:

gcDisable()
while true:
  gameLogic()
  renderFrame()
  gcStep(us = leftTime)
  sleep(restTime)

Type safe sets and arrays of enums

Often you want a mathematical set over values you defined yourself. Here’s how you do this in type-safe way:

type FakeTune = enum
  freeze, solo, noJump, noColl, noHook, jetpack

var x: set[FakeTune]

x.incl freeze
x.incl solo
x.excl solo

echo x + {noColl, noHook}

if freeze in x:
  echo "Here be freeze"

var y = {solo, noHook}
y.incl 0 # Error: type mismatch

You can’t accidentally add a value of another type. Internally the set works as an efficient bitvector.

The same is possible with arrays, indexing them with your enum:

var a: array[FakeTune, int]
a[freeze] = 100
echo a[freeze]

Unified Call Syntax

This is just syntactic sugar, but it’s definitely nice to have. In Python I always forget whether len and append are functions or methods. In Nim I don’t have to remember, because there is no difference. Nim uses Unified Call Syntax, which has now also been proposed for C++ by Herb Sutter and Bjarne Stroustrup.

var xs = @[1,2,3]

# Procedure call syntax
add(xs, 4_000_000)
echo len(xs)

# Method call syntax
xs.add(0b0101_0000_0000)
echo xs.len()

# Command invocation syntax
xs.add 0x06_FF_FF_FF
echo xs.len

Good performance

It’s easy to write fast code in Nim, as can be seen in the Longest Path Finding Benchmark, in which Nim is competing with some cute code.

I made some measurements on my machine when the benchmark was first published (Linux x86-64, Intel Core2Quad Q9300 @2.5GHz, state of 2014-12-20):

Lang	Time [ms]	Memory [KB]	Compile Time [ms]	Compressed Code [B]
Nim	1400	1460	893	486
C++	1478	2717	774	728
D	1518	2388	1614	669
Rust	1623	2632	6735	934
Java	1874	24428	812	778
OCaml	2384	4496	125	782
Go	3116	1664	596	618
Haskell	3329	5268	3002	1091
LuaJit	3857	2368	-	519
Lisp	8219	15876	1043	1007
Racket	8503	130284	24793	741

Compressed code size with gzip -9 < nim.nim | wc -c. Removed unused functions in Haskell. Compile times are clean compiles, if you have a nimcache with the standard library precompiled it’s only 323 ms for Nim.

Another small benchmark I did, calculating which numbers of the first 100 million are prime, in Python, Nim and C:

Python (runtime: 35.1 s)

def eratosthenes(n):
  sieve = [1] * 2 + [0] * (n - 1)
  for i in range(int(n**0.5)):
    if not sieve[i]:
      for j in range(i*i, n+1, i):
        sieve[j] = 1
  return sieve

eratosthenes(100000000)

Nim (runtime: 2.6 s)

import math

proc eratosthenes(n): auto =
  result = newSeq[int8](n+1)
  result[0] = 1; result[1] = 1

  for i in 0 .. int sqrt(float n):
    if result[i] == 0:
      for j in countup(i*i, n, i):
        result[j] = 1

discard eratosthenes(100_000_000)

C (runtime: 2.6 s)

#include <stdlib.h>
#include <math.h>
char* eratosthenes(int n)
{
  char* sieve = calloc(n+1,sizeof(char));
  sieve[0] = 1; sieve[1] = 1;
  int m = (int) sqrt((double) n);

  for(int i = 0; i <= m; i++) {
    if(!sieve[i]) {
      for (int j = i*i; j <= n; j += i)
        sieve[j] = 1;
    }
  }
  return sieve;
}

int main() {
  eratosthenes(100000000);
}

Compile to JavaScript

You can compile Nim to JavaScript, instead of C. This allows you to write the client as well as the server directly in Nim. Let’s make a small visitor counter on the server that gets displayed in the browser. This is our client.nim:

import htmlgen, dom

type Data = object
  visitors {.importc.}: int
  uniques {.importc.}: int
  ip {.importc.}: cstring

proc printInfo(data: Data) {.exportc.} =
  var infoDiv = document.getElementById("info")
  infoDiv.innerHTML = p("You're visitor number ", $data.visitors,
    ", unique visitor number ", $data.uniques,
    " today. Your IP is ", $data.ip, ".")

We define a Data type that we use to pass data from the server to client. The printInfo procedure will be called with this data and display it. Compile this with nim js client. The result javascript file ends up in nimcache/client.js.

For the server we need to get the Nimble package manager and run nimble install jester. Now we can use the Jester web framework and write our server.nim:

import jester, asyncdispatch, json, strutils, times, sets, htmlgen, strtabs, httpcore

var
  visitors = 0
  uniques = initSet[string]()
  time: TimeInfo

routes:
  get "/":
    resp body(
      `div`(id="info"),
      script(src="/client.js", `type`="text/javascript"),
      script(src="/visitors", `type`="text/javascript"))

  get "/client.js":
    const result = staticExec "nim -d:release js client"
    const clientJS = staticRead "nimcache/client.js"
    resp clientJS

  get "/visitors":
    let newTime = getTime().getLocalTime
    if newTime.monthDay != time.monthDay:
      visitors = 0
      init uniques
      time = newTime

    inc visitors
    let ip =
      if request.headers.hasKey "X-Forwarded-For":
        request.headers["X-Forwarded-For", 0]
      else:
        request.ip
    uniques.incl ip

    let json = %{"visitors": %visitors,
                 "uniques": %uniques.len,
                 "ip": %ip}
    resp "printInfo($#)".format(json)

runForever()

The server delivers the main website. It also delivers the client.js, by compiling and reading the client.nim at compile time. The logic is in the /visitors handling. Compile and run with nim -r c server and open http://localhost:5000/ to see the code in effect.

You can see our code in action on the Jester-generated site or inline here:

You don't have JavaScript enabled or something went wrong.

Final words

I hope I could pique your interest in in the Nim programming language.

Note that the language is not completely stable yet. Especially with the more obscure features you may run into bugs. But on the bright side, Nim 1.0 is supposed to be released within the next 3 months! So now is the perfect time to get started with learning Nim.

Bonus: Since Nim compiles to C and only depends on the C standard library you can deploy it nearly everywhere, including x86-64, ARM and Intel Xeon Phi accelerator cards.

For comments use Reddit, Hacker News or ask the Nim community directly on IRC (#nim on freenode). You can reach me personally at dennis@felsing.org.

Thanks to Andreas Rumpf and Dominik Picheta for proof-reading this post.

Nim Adventures

2014-07-13T00:00:00-04:00

To learn some Nim I held a talk about it at the GPN14 (in German, Slides).

Afterwards I started solving Rosetta Code tasks in Nim to get a better feel for the language and standard library. That also made me discover some rough edges in the language, but luckily the community, albeit small, is active and competent. All the small code pieces I wrote are now on Github too.

What I noticed is that most problems are as easy to solve as in Python, but much more performant. I’m now more confident that this language is the right choice for writing HookRace in.

Some highlights:

Insertion Sort

Once in Python:

def insertion_sort(l):
    for i in xrange(1, len(l)):
        j = i-1 
        key = l[i]
        while (l[j] > key) and (j >= 0):
           l[j+1] = l[j]
           j -= 1
        l[j+1] = key

and in Nim:

proc insertSort[T](a: var openarray[T]) =
  for i in 1 .. <a.len:
    let value = a[i]
    var j = i
    while j > 0 and value < a[j-1]:
      a[j] = a[j-1]
      dec j
    a[j] = value

Rank languages by popularity

The standard library is extremely useful, providing an HTTP client, JSON parser, regular expressions, string utils and algorithms, which I use to create a ranking of the popularity of languages on Rosetta Code:

import httpclient, json, re, strutils, algorithm, future

const
  langSite = "http://www.rosettacode.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Programming_Languages&cmlimit=500&format=json"
  catSize = "http://www.rosettacode.org/w/index.php?title=Special:Categories&limit=5000"
let regex = re"title=""Category:(.*?)"">.+?</a>.*\((.*) members\)"

var langs = newSeq[string]()
for l in parseJson(getContent(langSite))["query"]["categorymembers"]:
  langs.add(l["title"].str.split("Category:")[1])

var ranks = newSeq[tuple[lang: string, count: int]]()
for line in getContent(catSize).findAll(regex):
  let lang = line.replacef(regex, "$1")
  if lang in langs:
    let count = parseInt(line.replacef(regex, "$2").strip())
    ranks.add((lang, count))

ranks.sort((x, y) => cmp[int](y.count, x.count))
for i, l in ranks:
  echo align($(i+1), 3), align($l.count, 5), " - ", l.lang

Calling a C function

proc printf(formatstr: cstring) {.header: "<stdio.h>", varargs.}

var x = "foo"
printf("Hello %d %s!\n", 12, x)

Wrapping a shared C library

when defined(windows):
  const imglib = "imglib.dll"
elif defined(macosx):
  const imglib = "imglib.dylib"
else:
  const imglib = "imglib.so"

proc openimage(s: cstring): cint {.importc, dynlib: imglib.}

echo openimage("foo")

Quine

A quine is a program that prints its own source code. This Nim program does that once at compiletime and once at runtime:

const x = "const x = |const y = x[0..9]&34.chr&x&34.chr&10.chr&x[11..100]&10.chr&x[102..115]&10.chr&x[117 .. -1]|static: echo y|echo y"
const y = x[0..9]&34.chr&x&34.chr&10.chr&x[11..100]&10.chr&x[102..115]&10.chr&x[117 .. -1]
static: echo y
echo y

What is HookRace?

2014-07-07T00:00:00-04:00

I have been running DDraceNetwork for 1 year, which started out as a little server of a Teeworlds modification called DDRace. It has turned into the biggest project within Teeworlds, offering servers in 6 countries with thousands of players and dozens of mappers. We also offer a custom server and client (Windows, Linux, OS X, Android).

Now it’s time to start something new. I’m working on a new game called HookRace, which will be the successor to DDraceNetwork. It will be its own game, not based on Teeworlds.

Some of the plans I have so far for HookRace:

Vector Graphics (SVG) ingame for a sharp image on any screen
Whole physics is executed at the client as well as the server, for perfect prediction
No lags with high player numbers
Accounts to prevent faking

As the programming language for HookRace, I chose Nim:

# Game of Life in terminal
import os, strutils, math

randomize()

var w, h: int
if paramCount() >= 2:
  w = parseInt paramStr 1
  h = parseInt paramStr 2
if w <= 0: w = 30
if h <= 0: h = 30

# Iterate over fields in the universe
iterator fields(a = (0,0), b = (h-1,w-1)) =
  for y in a[0]..b[0]:
    for x in a[1]..b[1]:
      yield (y,x)

# Create a sequence with an initializer
proc newSeqWith[T](len: int, init: T): seq[T] =
  result = newSeq[T] len
  for i in 0 .. <len:
    result[i] = init

# Initialize
var univ, univNew = newSeqWith(h, newSeq[bool] w)

for y,x in fields():
  if random(10) < 1: univ[y][x] = true

while true:
  # Show
  stdout.write "\e[H"
  for y,x in fields():
    stdout.write if univ[y][x]: "\e[07m  \e[m" else: "  "
    if x == 0: stdout.write "\e[E"
  stdout.flushFile

  # Evolve
  for y,x in fields():
    var n = 0
    for y1,x1 in fields((y-1,x-1), (y+1,x+1)):
      if univ[(y1+h) mod h][(x1 + w) mod w]:
        inc n

    if univ[y][x]: dec n
    univNew[y][x] = n == 3 or (n == 2 and univ[y][x])
  swap univ, univNew

  sleep 200

Nim is a rather new language, and it took me some time to choose it. What I love about the language:

Python-like syntax
Statically typed, compiled
High performance (same ballpark as C/C++)
Garbage Collector that can be controlled for soft real-time
Produces executables without dependencies (compiles to C)
Easily interfaces C libraries (compiles to C)
Clean and powerful metaprogramming

On this blog I will report about my progress in the development of HookRace and my current Nim adventures.