DDNet Posts · HookRace Blog

Port Scan with Spoofed IP Addresses

2024-03-19T00:00:00-04:00

Or how to make naïve hosters shut down your victim’s server.

Do you want to bring down your victim’s game servers? Do they host their game servers on cheap VPSes at hosters who are not that experienced with networking? Is running an actual DoS against the server too expensive and illegal for you?

Just grab a server for yourself from a questionable hoster with IP address spoofing allowed on their network. Next run a port scan using nmap (man page) with a spoofed source address (-S or -D for multiple decoys) using your victim’s IP address(es). As the target of the port scan choose a university with a cybersecurity department that likes to send security incident reports to abuse mail addresses. To the target university of the port scan it will look like the scan came from your victim, since you are spoofing their IP address, so they will send a serious sounding email to the victim’s hoster:

As a consequence your victim’s VPS will most likely be shut down for a few hours or even days. To prevent this your victim will try to block all of their incoming and outgoing TCP traffic on their VPS, and also block the reporting university’s IP range entirely:

$ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  147.251.0.0/16       anywhere
DROP       tcp  --  anywhere             anywhere             tcp ctstate NEW multiport dports  !27685,6546,ssh

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  anywhere             147.251.0.0/16

And yet you can rerun your port scan against the university’s network every day, which will trigger automatic abuse mails to the victim’s hoster, who will still trust these abuse mails more than the victim, since the hoster is unlikely to be able to monitor the network traffic of the VPS to check if those packets actually originated there. At some point the hoster will be so annoyed that they simply terminate the VPS and throw out the customer, victory!

Congratulations, you have brought down some small-fish game servers in the DDNet community and made them lose the remaining money they had already put into the hoster! Now you can collect the disgruntled players on your own servers, post your freedom-fighting messages on Discord, and feel proud of having some hard-earned power!

In the next post we will continue in this vein and show you the equally satisfying techniques to steal candy from a baby.

P.S.: If anyone has better suggestions than manually sending the same explanations and this link to my server hosters every time they get such an abuse report, please tell me.

Update on DoS Attacks against our Online Game

2022-05-16T00:00:00-04:00

In my previous post 8 months ago I described how our open source online game DDraceNetwork has been suffering under DoS attacks for about 8 years, basically since its inception. Recently the attacks have gotten much worse, forcing us to work on further approaches. Since many players made suggestions recently, I’m writing this blog post to summarize what we are attempting and to ask for help again.

These traffic graphs are from two of the servers we are running, note the logarithmic x-axis. Each spike represents an incoming DoS attack, as you can see some of them last for nearly a day.

Recently the attacks have been relatively weak in terms of incoming bandwidth, using spoofed IP addresses imitating our UDP-based connection process.

At first the CPU gets overloaded since the server suddenly has to try and handle hundreds of thousands of connection attempts per second. Since our game servers are mostly running on cheap VPSes and each game server runs single-threaded, it is quite easy to overload a system in this manner.

HTTPS-based Whitelist

To prevent spoofing we collect all players’ IP addresses and whitelist those. Since we also develop the game client, we can modify the client to connect to a server via HTTPs for this whitelisting. The iptables rules and ipset setup on the game servers for this whitelist look something like this:

ipset create official iphash
ipset create whitelist-ip iphash
iptables -N serverinfo
iptables -A serverinfo -m hashlimit --hashlimit-above 40/s --hashlimit-mode dstport --hashlimit-name si_dstport -j DROP
iptables -N game
iptables -A game -m set --match-set official src -j ACCEPT
iptables -A game -m set --match-set whitelist-ip src -m u32 --u32 "38=0x67696533" -j serverinfo
iptables -A game -m set --match-set whitelist-ip src -m u32 --u32 "38=0x66737464" -j serverinfo
iptables -A game -m set --match-set whitelist-ip src -j ACCEPT
# Still allow non-whitelisted players when there is no attack
iptables -A game -m limit --limit 10000 -j ACCEPT
iptables -A game -j DROP
iptables -A INPUT -p udp -m udp --dport 8000:9000 -j game

To keep updating the whitelist we run a simple script:

#!/bin/sh
rm -f whitelist whitelist.old
touch whitelist.old
while true; do
  curl -o whitelist $WHITELIST_URL
  LC_ALL=C comm -23 whitelist whitelist.old | while read line; do
    ipset add whitelist-ip $line
  done
  mv whitelist whitelist.old
  sleep 20
done

About 20 seconds after starting DDNet client the players are now being whitelisted and can join servers which are under a spoofed DoS attack. We are employing this on the servers which are commonly attacked.

After a few days the ipset runs out of space unfortunately. Since we have ~25k unique players daily, the default maximum element size of 65536 of ipset is quite tight. To work around this we’ll have to increase the limit or remove older entries automatically.

In order to make this work reliably we had to disable IPv6 for the HTTPS access, since our game servers are IPv4 only at the moment and collecting IPv6 addresses doesn’t help.

An unfortunate realization was that some ISPs don’t provide fixed IPv4 addresses anymore, but instead tunnel their native IPv6 traffic through different IPv4 addresses, based on protocol and port. So we get one IP address in the HTTPS-based whitelist, but another when connecting to the gameserver via UDP. At least one Israeli ISP is doing this.

Dedicated Servers

For some of the servers this whitelisting is not enough, since the hosters’ DoS protection kicks in and starts aggressively blocking UDP traffic. This includes legitimate traffic and results in all players timing out on the game servers. I have tried talking to a few of the hosters about this, but it is by design since they don’t support our game protocol, and wish to protect their own infrastructure as well as other customers. At least they are not kicking us out for now, which has happened in the past with a few hosters after multiple DoS attacks against us impacting their other customers.

So for a few locations we have upgraded to dedicated servers, where the hoster cares less about us exhausting in the incoming network connection and our CPUs for hours at a time. Unfortunately it is difficult to find hosters that provide more than 1 Gbit/s network bandwidth and still fall into our budget of 100 € / month.

The attackers of course notice that these servers can’t be downed as easily, so they switch to different kinds of attacks, exhausting our measly 2 Gbit/s network.

Small Hosting Companies

Some small hosting companies in Europe offered to help us by providing a free server with custom DoS protection. We accepted two such offers, but didn’t have much success with either of them.

For the first we never managed to get legitimate traffic to get through their firewall.

For the second everything seemed to work fine when there was no attack, but during DoS attacks some player packets would get misflagged all the time. Additionally the network stack had problems and caused our MariaDB-based SQL connection to time out. Unfortunately we didn’t handle this error properly in our code and ended up losing legitimate player ranks.

In the end we didn’t continue using either of these hosters, but spent a few hours trying to set up the firewall with them. I had similar experiences with every paid hoster which offered a custom DoS protection.

DoS Protection Companies

There are of course also larger companies providing custom DoS protection for UDP-based applications. One of them offered to help us out for free with our problem, and I’ve been in touch with them since the last post. Unfortunately everything is moving very slowly, and many of my requests to move things forward over the weeks have not been answered, so I guess the fact that we are unable to pay thousands of € per month matters after all. So far no custom DoS protection is available here, but I’ll keep pinging them every few weeks.

Proxy Servers

Instead of exposing our real game servers directly to the players, for the future we are considering a proxy server that sits inbetween the game server and player. This would allow us to run multiple proxies for subsets of the players and attempt to isolate the impact the attacker can have.

This might also allow us to proxy Steam players’ traffic through Steam Datagram Relay (SDR), although more work would be required to implement that. Someone from Valve reached out to us after the previous post, and we might follow up on this once the proxy server implementation is ready. It is really cool that Valve is offering help here, even for a free game like ours.

We can’t entirely switch to SDR through Steam’s servers since we want DDNet to stay playable as an open source game. As a solution we can provide a regular open source proxy for open source players, and a separate SDR-based proxy for Steam players.

IPv6 Servers

A longshot idea is using IPv6-only servers. We haven’t tried this yet, maybe the IP spoofing capabilities of our attackers are lower on IPv6. This could also be combined well with the proxy servers by providing an additional ipv6-only proxy server.

Running all these proxy servers for each location will definitely add cost, but it’s probably our best path forward right now. The setup would look something like this for each Server location:

digraph D {
  node [shape="box"];
  "OS Player 1" -> "IPv4 Proxy" -> "Game Server";
  "OS Player 2" -> "IPv4 Proxy";
  "OS Player 3" -> "IPv6 Proxy" -> "Game Server";
  "OS Player 4" -> "IPv6 Proxy";
  "Steam Player 1" -> "SDR Proxy" -> "Game Server";
  "Steam Player 2" -> "SDR Proxy";
}

Conclusion

If you have any suggestions, or dealt with a similar problem before, we’d be interested to hear from you. You can reach me on dennis@felsing.org as well as on the DDNet Discord as deen#5910 or on IRC (deen in #ddnet on Quakenet).

DoS Attacks against our Online Game

2021-09-27T00:00:00-04:00

DDraceNetwork is an open source online game I’ve been running since 2013 with a community of volunteers. The game is available for free, I’m hosting servers for it in many countries around the world so that we have trusted official ranks. The servers are paid for by donations, which I stop collecting once the cost of the servers for the current year is covered. I wrote about DDNet in a previous post in 2016 and also a bit about the game history in 2013.

Pretty much since the beginning we have been suffering from DoS attacks against the servers. Since the Steam release in 2020 the player number has increased significantly, so that we have about 1300 players playing on average.

Based on the rising player number the urge to deal with the DoS problem is larger than ever. A few months ago for example we organized a Tournament for everyone in the community to participate. Unfortunately the event was the continuous target of DoS attacks for multiple hours, with players fleeing from one server to the next, trying to find a safe refuge. These bandwidth graphs for our servers in Netherlands and Germany respectively illustrate the problem:

Since DDNet only runs online and you feel every small network latency it is predestined as a target for DoS attacks. Outside of attacks our servers don’t need to be particularly powerful for regular gameplay, 1 CPU core can support 150 players concurrently, memory consumption is minimal. So the total we are paying for our ~23 servers around the world is only ~2300 € for the year 2021.

What we have tried

The kinds of attacks we receive are varied. There are generic reflection and amplification attacks, as well as spoofed game specific attacks of considerable strength. Some hosters have thrown us out following attacks impacting their other customers, and most hosters will blackhole our IP address for multiple hours before it gets to that.

There is even a Wireshark dissector for DDNet/Teeworlds available, written in Rust:

Some components of our system were relatively easy to protect: The database server has been moved to a secret IP address, and the game servers will store ranks in a local SQLite file if the main MySQL-based database is not available.

The web server hosting DDNet.tw and other HTTPS-based components like the map downloader and updater are now reachable through Cloudflare only, and have received no attacks since then.

For the individual server infos the client currently has to communicate with each game server by UDP, thus revealing its own IP address without having connected to a server. Since one of the known attackers is running their own DDNet server, they can use this method to collect legitimate player IP addresses and spoof them in their attacks. To prevent this heinrich5991 implemented a centrally provided HTTPS-based server info. This means that every player only has to access the JSON file containing all server information, instead of previously receiving this information via UDP from each server directly.

To prevent our own servers being used for amplification attacks we limit the server info packets per IP address in our server setup script:

iptables -t raw -A PREROUTING -p udp -j NOTRACK
iptables -t raw -A OUTPUT -p udp -j NOTRACK
iptables -N serverinfo
iptables -A INPUT -p udp -m u32 --u32 "38=0x67696533" -j serverinfo
iptables -A INPUT -p udp -m u32 --u32 "38=0x66737464" -j serverinfo
iptables -A serverinfo -m hashlimit --hashlimit-above 1000/s --hashlimit-burst 2500 --hashlimit-mode dstport --hashlimit-name si_dstport -j DROP
iptables -A serverinfo -m hashlimit --hashlimit-above 20/s --hashlimit-burst 100 --hashlimit-mode srcip --hashlimit-name si_srcip -j DROP
iptables-save > /etc/iptables.up.rules

Unfortunately even sending a few packets per second is considered a lot by some people, so with the spoofed server info requests coming in, we get a lot of complaints for the “port scanning” that we seem to be doing. I have to react quickly to those abuse mails in order to prevent our servers from being shut down, having to explain the situation and blocking any traffic from the IP range in question. An example mail:

On 2021-04-19T14:58+0200, abuse@hetzner.com wrote:

On 18 Apr 12:03, abuse-out@X.com wrote:

The address 49.12.97.180 from Your network tried to log in to our network using Port […]

Below You will find a listing of the dates and times the incidents occured as well as the attacked IP-Addresses. This is a matter of concern for us and continued tries might result in legal action. If the machine was victim to a hack take it offline, repair the damage and use better protection next time. The times included are in Central European Time (CET).

[…]

Regards,

X

Dear Mr Dennis Felsing,

We have received information regarding spam and/or abuse from abuse-out@X.com Please take all necessary measures to avoid this in the future.

We also request that you send a statement within 24 hours to us and to the person who filed the complaint. This response should contain information about how this could have happened and what you intend to do about it.

How to proceed:

Solve the issue

Send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=X

Send a response by email to the person who filed the complaint

The statement will be checked by a staff member who will then coordinate any further proceedings. If you fail to comply within the stated deadline, the IP may be blocked.

Important note: When replying to us, please leave the abuse ID [AbuseID:X] unchanged in the subject line.

Kind regards

X

Hi Mr. X,

The server in question (49.12.97.180) only responds to incoming requests. We received an incoming DoS attack with spoofed IP addresses, so the server replied to those. We already limit the number of packets per server that we respond to. For the future I have blocked all packets from X/24 of reaching the server, so the server also won’t send any packets to them.

Best regards

Dennis Felsing

When players get a timeout ingame, they can reconnect later and keep playing from where they were before. We tried detecting a DoS attack and prolonging the timeout protection to 1 hour, but this lead to the problem of servers being full of characters having a timeout, making it impossible for new players to join, even those trying to reclaim their timed out character. Additionally teams can use the save functionality to save their progress and continue it at any later time, or even on another server.

This doesn’t help when a VPS gets nullrouted for hours or even days at a time due to attacks. Many hosters offer “DDoS protection” for their servers, but so far none worked well. Some block all UDP traffic during an attack, which is clearly suboptimal for a UDP-based game. Others try to train their DDoS detection a bit to detect legitimate traffic, but some players and some packets of all players will still be considered as part of the attack and get blocked. The attackers can even use the DDoS protection and spoof the collected IP addresses of real players to get specific player IP addresses blocked. As a reaction of that I then have to discuss with the hoster for a long time that the players probably are not part of a botnet, but being spoofed.

Instead of cheap VPS servers we have tried getting dedicated servers at larger European hosters like OVH, Hetzner, ihor and NFOrce. The idea is that we have exclusive resources, so the chances of us impacting other customers is lower, and thus we won’t get nullrouted so easily. Largely this works, but the available network bandwidth (usually 1-10 Gbit/s) as well as CPU usage become the limit. Each individual game server uses a single thread to handle all network packets, so it is still relatively easy and cheap for attackers to overload.

Cloud-based servers at AWS or Google Cloud perform well, even during attacks, but the bandwidth cost is exorbitant. We could consider using them for short periods during tournaments instead of permanently though.

What we can still try

So far we have only tried 10 Gbit/s dedicated servers. Maybe the reason that the Cloud providers function so well during DoS attacks is that the actual servers the virtual machines run on have even larger bandwidth available. So we could still try to go to 20-25 Gbit/s for some dedicated servers, but only few cheap hosters offer this and what they charge for it is a lot for us. An example would be an additional 50 € booking fee and 100 € more per month for 20 Gbit/s at NFOrce

The network handling of the game servers can be switched to run multithreaded, so that we can at least handle the packets of unconnected players faster. This doesn’t help with the really cheap servers, and the attacker could just increase the attack strength beyond what we can handle with our usual 2-4 available cores. This already happens when multiple game servers are attacked at once, so it seems like it might not help much.

Instead of improving the game servers, we could have a Linux kernel based firewall using Express Data Path (XDP) for higher performance. Other server hosters, namely noby and ReiTW, had success with this approach, but for us this hasn’t helped much yet, probably since we are still out of bandwidth and CPU.

So the next logical step would be not to have this kind of firewall on our rented virtual machines, but as part of the dedicated DoS solutions that the hosters are providing. Unfortunately so far no hoster was willing to add it, except for a large fee. Hosting a more popular online game would make this easier, maybe we should just imitate another game’s network packets to benefit from the improved DoS protection support.

Instead of another UDP-based game’s network packets we also have a way to communicate using Websockets instead. Potentially Cloudflare or other web-oriented DoS protections could protect that well. Unfortunately Websockets are TCP-based and thus not perfectly suited for a fast-paced game where it makes no sense to resend out-of-date state updates from server to client.

Of course there are proper solutions that offer DoS protection for UDP-based games, such as Cloudflare Gaming or Akamai Gaming, but their costs start at thousands of € per month. Thus we clearly can’t afford them.

Instead of technical solutions, we could try working with the police. We know the real name of the main DoS attacker, but there is no way to link him to the attacks from the data we have. Even then, this might not help much. I have previously reported a DoS attacker to the police, since he was kind enough to send a message claiming an attack as his own from his real phone number. But since the attacker was a minor and we are a free online game and thus have no measurable economic damage, the public prosecutor left it at a sternly worded warning.

The Unknown

I wrote most of this post back in May when we were facing large-scale attacks the last time. Since then not much happened, so I didn’t want to bring any unwanted attention to our situation. Now in September the attacks seem to be starting again with a long downtime on all our servers today, so I decided to finish this up and post it anyway. Let’s hope that this brings more positive attention than negative. After all, I just showed exactly how good of a DoS target we are.

Discussion on Hacker News.

Theme Switcher in Nginx

2019-01-09T00:00:00-05:00

Some people really liked the dark DDNet theme for Halloween by Soreu, so we decided to keep it possible to use the default bright or the dark theme.

Thanks to xse we got a JavaScript based theme switcher. After some improvements I finally I switched it away from JavaScript entirely and finally am also using it on this blog with an OLED friendly dark theme.

The final result is quite simple, and only requires static HTML, a cookie for the theme and Nginx for setting and returning the correct CSS file for it. You can try it out by clicking on the Switch Theme button at the top of this page, which just redirects you to /switch-theme/.

Doing the CSS decision server-side instead of in JavaScript has the advantage that you don’t get any flicker on rendering, no matter what theme you chose.

The entire nginx logic is:

location /switch-theme/ {
  if ($http_cookie ~* "user_theme=/public/css-dark.css") {
    add_header Set-Cookie "user_theme=/public/css-empty.css;path=/;domain=hookrace.net";
    return 302 $http_referer;
  }
  add_header Set-Cookie "user_theme=/public/css-dark.css;path=/;domain=hookrace.net";
  return 302 $http_referer;
}

location /public/css-dark.css {
  if ($http_cookie ~* "user_theme=/public/css-dark.css") {
    return 302 /public/css/css-dark.css;
  }
  return 302 /public/css/css-empty.css;
}

Note that the /switch-theme/ location sends the user back to the $http_referer. /public/css-dark.css is statically included on each page, but what it does depends on the cookie value.

The 302 Moved Temporarily response makes the client send another request, adding another roundtrip to the page rendering. If the CSS is small enough you can just pull it directly into the Nginx config:

location /public/css-dark.css {
  if ($http_cookie ~* "user_theme=/public/css-dark.css") {
    return 200 "body {color: #fff; background-color: #000;} .page-title, .post-title, .post-title a, h1, h2, h3, h4, h5, h6 {color: #ccc;} .masthead-title a {color: #bbb;}";
  }
  return 200 ""; 
}

One side effect of this entire approach is that the CSS file can’t be cached anymore though, otherwise we’d end up showing the old theme when it gets switched. I’m sure this could be optimized somehow to tell the browser to only fetch the CSS file again after the theme was switched. But for my site the current performance is good enough.

Also note that If Is Evil in Nginx, so if you abuse features like this you might run into trouble.

Writing a 2D Platform Game in Nim with SDL2

2016-06-14T00:00:00-04:00

Japanese Translation

In this article we’re going to write a simple 2D platform game. You can also consider this as a tutorial for game development with SDL2 in Nim.

We will read in user input, display graphics and a tile map, and simulate simple 2D physics with collision detection and handling. Afterwards we will implement simple camera movement and game logic. To display some information we will render texts and develop a caching mechanism for said text rendering.

The final result will be a binary file that requires only SDL2 and can be easily distributed, perfect for games. If you’re on Linux we will also present a simple way to cross-compile Nim programs for Windows.

For the sake of simplicity we’re going to use the familiar graphics from DDNet and Teeworlds, with the end result of this tutorial looking like this:

We’re going to follow along with the development throughout this article with illustrative images and videos, but the best way to learn is if you follow along yourself by implementing the steps from this article. The code is purposefully kept simple and easy to extend so that you can play around with it and try out all kinds of changes to get an intuitive understanding. At the end of every section there is a link to its full source code.

The iterations of the code of this article and the final result are available in a repository on GitHub. The resulting binaries can be downloaded here: Win64, Win32, Linux x86_64, Linux x86

Preliminaries

For this post we require:

The Nim programming language and its package manager Nimble
SDL 2, SDL_image 2, SDL_ttf 2 (all for developers)
Nim SDL2 wrapper and strfmt, which can be installed using Nimble

On a unixoid system like Linux or Mac OS X the installation looks something like this:

# Debian / Ubuntu
$ sudo apt-get install git libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev
# Arch Linux
$ pacman -S git sdl2 sdl2_image sdl2_ttf
# Homebrew on OS X
$ brew install git sdl2 sdl2_image sdl2_ttf
# FreeBSD
$ pkg install git sdl2 sdl2_image sdl2_ttf

$ wget http://nim-lang.org/download/nim-0.14.2.tar.xz
$ tar xvf nim-0.14.2.tar.xz
$ cd nim-0.14.2
$ make -j4
$ echo 'export PATH=$HOME/nim-0.14.2/bin:$PATH' >> ~/.profile
$ source ~/.profile

$ git clone https://github.com/nim-lang/nimble.git
$ cd nimble
$ nim -d:release c -r src/nimble install
$ echo 'export PATH=$HOME/.nimble/bin:$PATH' >> ~/.profile
$ source ~/.profile

$ nimble install sdl2 strfmt

Note that you also need a C compiler on your system, preferably GCC or Clang.

Instead of compiling Nim and Nimble from source code you could also use your package manager to install Nim and Nimble if they’re available in a recent version.

For setting up SDL2 on other platforms more extensive guides exist, as do for Nim and nimble.

1. First Running Program

If you have seen SDL2 programs written in C or C++ before, you will notice that what we’re doing in Nim is very similar. Actually Nim’s SDL2 wrapper is just a thin layer wrapping the original SDL2 interface from C. This has the advantage that what you learn from any SDL2 tutorial is applicable, but the disadvantage is that you end up with a bit more boilerplate than with a more high-level Nim library.

We start with exactly this boilerplate to initialize our window:

import sdl2

type SDLException = object of Exception

template sdlFailIf(cond: typed, reason: string) =
  if cond: raise SDLException.newException(
    reason & ", SDL error: " & $getError())

proc main =
  sdlFailIf(not sdl2.init(INIT_VIDEO or INIT_TIMER or INIT_EVENTS)):
    "SDL2 initialization failed"

  # defer blocks get called at the end of the procedure, even if an
  # exception has been thrown
  defer: sdl2.quit()

  sdlFailIf(not setHint("SDL_RENDER_SCALE_QUALITY", "2")):
    "Linear texture filtering could not be enabled"

  let window = createWindow(title = "Our own 2D platformer",
    x = SDL_WINDOWPOS_CENTERED, y = SDL_WINDOWPOS_CENTERED,
    w = 1280, h = 720, flags = SDL_WINDOW_SHOWN)
  sdlFailIf window.isNil: "Window could not be created"
  defer: window.destroy()

  let renderer = window.createRenderer(index = -1,
    flags = Renderer_Accelerated or Renderer_PresentVsync)
  sdlFailIf renderer.isNil: "Renderer could not be created"
  defer: renderer.destroy()

  # Set the default color to use for drawing
  renderer.setDrawColor(r = 110, g = 132, b = 174)

  # Game loop, draws each frame
  while true:
    # Draw over all drawings of the last frame with the default
    # color
    renderer.clear()
    # Show the result on screen
    renderer.present()

main()

We introduced an sdlFailIf template that checks a condition and if the condition is true, raises an SDLException with additional error information from SDL. In the main proc we initialize SDL2, and create a regular window and an accelerated 2D renderer. Error handling is done with the sdlFailIf proc that we introduced.

For now the game loop just clears the window and draws it every frame. If you have VSync enabled and your screen is set to 60 Hz the loop will be executed 60 times per second.

We can compile and run in the same step by executing nim -r c platformer, assuming you called the file platformer.nim. To compile with optimizations use nim -d:release -r c platformer. The result is a simple one-colored window:

We can exit our small program by pressing Ctrl-C in the terminal window. Unfortunately we can’t exit it in the game window itself yet, so let’s fix that.

Full code for section 1

2. User Input

First let’s add an Input type to store all the inputs we want to support, and store an array of Inputs in our game state object:

type
  Input {.pure.} = enum none, left, right, jump, restart, quit

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr

By choosing a ref type for the Game state type we have an easy way to prevent accidentally creating copies of it. By default only the garbage collected pointer to our Game object is passed around. The inputs field is an array mapping from Input to bool, signifying which input is currently pressed (true) or not (false). Creating a new game state object is trivial, we just create a new heap object for now and assign the SDL2 renderer that we will need later:

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

We don’t need to initialize the inputs field in any way as everything is initialized to binary null by default, which is exactly what we want: Every input is set to off in the start. If we didn’t initialize the renderer field it would be a null pointer and we would get into trouble if we accidentally dereference it.

The next thing we need is a procedure that maps keyboard scan codes to our recognized inputs:

proc toInput(key: Scancode): Input =
  case key
  of SDL_SCANCODE_A: Input.left
  of SDL_SCANCODE_D: Input.right
  of SDL_SCANCODE_SPACE: Input.jump
  of SDL_SCANCODE_R: Input.restart
  of SDL_SCANCODE_Q: Input.quit
  else: Input.none

Note that toInput returns Input.none for all undefined cases. We will use this behaviour to ignore unused keyboard inputs without the need for a branch in our code. You could easily recognize multiple scan codes to map to a single input.

We modify the game loop to react to keyboard inputs by calling our new handleInput proc. We also split out the rendering itself to keep the separation of concerns clear in the main proc:

proc handleInput(game: Game) =
  var event = defaultEvent
  while pollEvent(event):
    case event.kind
    of QuitEvent:
      game.inputs[Input.quit] = true
    of KeyDown:
      game.inputs[event.key.keysym.scancode.toInput] = true
    of KeyUp:
      game.inputs[event.key.keysym.scancode.toInput] = false
    else:
      discard

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Show the result on screen
  game.renderer.present()

var game = newGame(renderer)

# Game loop, draws each frame
while not game.inputs[Input.quit]:
  game.handleInput()
  game.render()

Now we can either press q or use the close button on the window to close the game. Other kinds of user input are just stored in the inputs array for now and we will be able to use it later.

Note that the inputs array is intentionally a simple array, so that access to it is as performant as can be. If we used a hash table or some other data structure such a guarantee would not be that easy to make. Simplicity can often be beneficial and make it easier to understand what is going on in the system you’re developing.

Full code for section 2

3. Displaying Graphics

If we want to do something of interest with those user inputs we need to start displaying something other than a blue sky.

We extend our game state object to also store the player texture as well as the current position and velocity, for which we use the basic2d module:

import basic2d

type
  Player = ref object
    texture: TexturePtr
    pos: Point2d
    vel: Vector2d

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    player: Player
    camera: Vector2d

We’re going to use Teeworlds’ default tee graphic for our player:

You can save this file as player.png to follow along. If you feel funny you can also select one of hundreds of skins from the DDNet Skin Database, for example:

Don’t forget to call your player graphic player.png if you want to use an alternative one.

First we have to load the player graphic, whichever one you decided to use, in our definition of newGame and initialize the Game and Player data structures:

import sdl2/image

proc restartPlayer(player: Player) =
  player.pos = point2d(170, 500)
  player.vel = vector2d(0, 0)

proc newPlayer(texture: TexturePtr): Player =
  new result
  result.texture = texture
  result.restartPlayer()

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer
  result.player = newPlayer(renderer.loadTexture("player.png"))

restartPlayer is used to reset the player to its start position. The loadTexture procedure loads the PNG image into memory as an SDL2 texture that we can store in the Game object.

We also shouldn’t forget to initialize the SDL2 image module in our main proc, similarly to the SDL2 initialization:

const imgFlags: cint = IMG_INIT_PNG
sdlFailIf(image.init(imgFlags) != imgFlags):
  "SDL2 Image initialization failed"
defer: image.quit()

We only need support for PNG files, otherwise we could also add JPEG files with const imgFlags: cint = IMG_INIT_PNG or IMG_INIT_JPG.

Next our task is to put this together nicely. Of course the intention of the flexible player images is that parts of the body can move independently, but for the sake of simplicity we will put them into fixed positions. A simple addition would be to make the feet move in a rotating motion depending on the horizontal position of the player. Another addition would be to make the eyes follow the mouse cursor.

proc renderTee(renderer: RendererPtr, texture: TexturePtr,
               pos: Point2d) =
  let
    x = pos.x.cint
    y = pos.y.cint

  var bodyParts: array[8, tuple[source, dest: Rect, flip: cint]] = [
    (rect(192,  64, 64, 32), rect(x-60,    y, 96, 48),
     SDL_FLIP_NONE),      # back feet shadow
    (rect( 96,   0, 96, 96), rect(x-48, y-48, 96, 96),
     SDL_FLIP_NONE),      # body shadow
    (rect(192,  64, 64, 32), rect(x-36,    y, 96, 48),
     SDL_FLIP_NONE),      # front feet shadow
    (rect(192,  32, 64, 32), rect(x-60,    y, 96, 48),
     SDL_FLIP_NONE),      # back feet
    (rect(  0,   0, 96, 96), rect(x-48, y-48, 96, 96),
     SDL_FLIP_NONE),      # body
    (rect(192,  32, 64, 32), rect(x-36,    y, 96, 48),
     SDL_FLIP_NONE),      # front feet
    (rect( 64,  96, 32, 32), rect(x-18, y-21, 36, 36),
     SDL_FLIP_NONE),      # left eye
    (rect( 64,  96, 32, 32), rect( x-6, y-21, 36, 36),
     SDL_FLIP_HORIZONTAL) # right eye
  ]

  for part in bodyParts.mitems:
    renderer.copyEx(texture, part.source, part.dest, angle = 0.0,
                    center = nil, flip = part.flip)

The exact numbers are not so important, they are just how the player is meant to be put together. With renderTee we define which body part is drawn at which position and in which order. Finally each of these body parts is drawn with the SDL2 renderer using copyEx.

Now drawing the tee in our game loop is simply a call to renderTee away:

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  # Show the result on screen
  game.renderer.present()

Finally we have some visual progress again, look at the player floating in the sky:

Full code for section 3

4. Tile Map

Now that we have a working rendering system for the player, we need a map to play in. This requires us to store a texture as well as a list of tiles:

type
  Map = ref object
    texture: TexturePtr
    width, height: int
    tiles: seq[uint8]

  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    player: Player
    map: Map
    camera: Vector2d

Each tile is defined to be a uint8, which means a value between 0 and 255 inclusively. Conveniently the tileset graphic for from Teeworlds have 16 × 16 = 256 tiles. We will use the grass tileset:

Download and save this image as grass.png.

To initialize the Map data structure we shall write a little parser that parses maps of this format:

0  0  0 78  0  0  0  0  0  0  0  0  0  0
5  0  0 78  0  0  0  0  0  0  0  0  0  0
21  0  0 78  0  0  0  0  0  0  0  0  0  0
21  0  0 78  0  0  0  0  0  0  0  0  4  5
21  0  0 78  0  0  0  0  0  0  0  0 20 21
21  0  0 78  0  0  0  0  0  0  0  0 20 21
21  0  0 78  0  0  4  5  0  0  0  0 20 21
21  0  0 78  0  0 20 21  0  0  0  0 20 21
38  0  0 78  0  0 22 38  0  0  0  0 22 38
49 16 16 16 16 16 48 49 16 16 16 16 48 49
52 52 52 52 52 52 52 52 52 52 52 52 52 52

Our goal in this section is for this map with the grass tileset to result in this rendering:

Each number denotes the tile that is chosen from the grass tileset. We will use this default.map for the rest of this article. Our parser is implemented in newMap and looks like this:

import strutils

proc newMap(texture: TexturePtr, file: string): Map =
  new result
  result.texture = texture
  result.tiles = @[]

  for line in file.lines:
    var width = 0
    for word in line.split(' '):
      if word == "": continue
      let value = parseUInt(word)
      if value > uint(uint8.high):
        raise ValueError.newException(
          "Invalid value " & word & " in map " & file)
      result.tiles.add value.uint8
      inc width

    if result.width > 0 and result.width != width:
      raise ValueError.newException(
        "Incompatible line length in map " & file)
    result.width = width
    inc result.height

We split the file by line as well as by word. Each number is parsed to an unsigned integer and checked whether it is in the valid range of 0..255. The final width and height are calculated from the line length and number of lines. Errors in the map data cause an exception to be thrown, quitting our game.

We have to extend newGame to initialize the map now:

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer
  result.player = newPlayer(renderer.loadTexture("player.png"))
  result.map = newMap(renderer.loadTexture("grass.png"),
    "default.map")

At this point we have the texture and the tiles of the map. What’s missing is actually rendering all of this:

const
  tilesPerRow = 16
  tileSize: Point = (64.cint, 64.cint)

proc renderMap(renderer: RendererPtr, map: Map, camera: Vector2d) =
  var
    clip = rect(0, 0, tileSize.x, tileSize.y)
    dest = rect(0, 0, tileSize.x, tileSize.y)

  for i, tileNr in map.tiles:
    if tileNr == 0: continue

    clip.x = cint(tileNr mod tilesPerRow) * tileSize.x
    clip.y = cint(tileNr div tilesPerRow) * tileSize.y
    dest.x = cint(i mod map.width) * tileSize.x - camera.x.cint
    dest.y = cint(i div map.width) * tileSize.y - camera.y.cint

    renderer.copy(map.texture, unsafeAddr clip, unsafeAddr dest)

This is similar to renderTee but uses fixed size parts of the texture. The texture is cut into tiles of size 64 × 64 pixels with 16 tiles per line. We iterate over each tile in the map and render the tile tileNr from our map texture with. Tile 0 is the air tile and is always empty so we don’t need to render it, which improves performance as typical maps are in large parts empty.

Finally we have to render our map in the main proc after rendering the player, so that it is put on top of the player:

proc render(game: Game) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)
  # Show the result on screen
  game.renderer.present()

For now we don’t have a moving camera, so we use the static game.camera to get a fixed rendering position. But for now we can’t even move, so we should finally do something with the user input and implement a simple physics model.

The end result of this section features our beautiful map rendering:

Full code for section 4

5. Physics and Collisions

For our game physics we decide to have 50 ticks per second. We will only calculate the next iteration of the game when a new tick has arrived, independent of whether our game runs at 60 fps or even 240 fps. Let’s add the ticks to our main proc:

import times

proc physics(game: Game) =
  discard

var
  startTime = epochTime()
  lastTick = 0

while not game.inputs[Input.quit]:
  game.handleInput()

  let newTick = int((epochTime() - startTime) * 50)
  for tick in lastTick+1 .. newTick:
    game.physics()
  lastTick = newTick

  renderer.render(game)

That’s our physics framework, but so far the physics does nothing. We can start by adding gravity:

proc physics(game: Game) =
  game.player.vel.y += 0.75
  game.player.pos += game.player.vel

Well, there goes our player. I guess we also need to be able to restart the player now, so that we can see this amazing animation again and again just by pressing r:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  game.player.vel.y += 0.75
  game.player.pos += game.player.vel

This looks exactly like the previous GIF on repeat, so you can imagine it or click on play a few times.

Moving left and right with a and d as well as jumping with space is now easy to implement:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  if game.inputs[Input.jump]:
    game.player.vel.y = -21

  let direction = float(game.inputs[Input.right].int -
                        game.inputs[Input.left].int)

  game.player.vel.y += 0.75
  game.player.vel.x = clamp(
    0.5 * game.player.vel.x + 4.0 * direction, -8, 8)

  game.player.pos += game.player.vel

The specific values are determined by trial and error. You can change them around if you have other preferences. Note that we don’t set the player position directly, instead we modify the velocity vector and add that to the position. This is important for collision detection.

While I tried to move around as if the walls and ground had an effect, I probably failed at deceiving you and you noticed that we still don’t have any collision detection and handling. This code is largely adapted from Teeworlds. It works by checking for horizontal and vertical collisions with a tile in moveBox, which manipulates the player’s position based on the passed velocity vector.

When a collision occurs the player is moved just out of the tile in the right direction. For the sake of simplicity in isSolid we consider every tile other than air, start, finish a solid block. Floating point player positions are converted to indices in the tile map.

import math

type
  Collision {.pure.} = enum x, y, corner

const
  playerSize = vector2d(64, 64)

  air = 0
  start = 78
  finish = 110

getTile reads a tile from a specified position of the map, making sure not to over- or underflow:

proc getTile(map: Map, x, y: int): uint8 =
  let
    nx = clamp(x div tileSize.x, 0, map.width - 1)
    ny = clamp(y div tileSize.y, 0, map.height - 1)
    pos = ny * map.width + nx

  map.tiles[pos]

isSolid determines whether the player can collide with a tile. As we said every tile other than air, start and finish are able to be collided with:

proc isSolid(map: Map, x, y: int): bool =
  map.getTile(x, y) notin {air, start, finish}

proc isSolid(map: Map, point: Point2d): bool =
  map.isSolid(point.x.round.int, point.y.round.int)

A player is on the ground when there is a block below either side of its feet:

proc onGround(map: Map, pos: Point2d, size: Vector2d): bool =
  let size = size * 0.5
  result =
    map.isSolid(point2d(pos.x - size.x, pos.y + size.y + 1)) or
    map.isSolid(point2d(pos.x + size.x, pos.y + size.y + 1))

Meanwhile testBox considers the player as an axis aligned boundary box, and tells us if the player is stuck inside of any solid walls:

proc testBox(map: Map, pos: Point2d, size: Vector2d): bool =
  let size = size * 0.5
  result =
    map.isSolid(point2d(pos.x - size.x, pos.y - size.y)) or
    map.isSolid(point2d(pos.x + size.x, pos.y - size.y)) or
    map.isSolid(point2d(pos.x - size.x, pos.y + size.y)) or
    map.isSolid(point2d(pos.x + size.x, pos.y + size.y))

moveBox now tries to apply the velocity vector vel to the player’s position pos. When this causes a collision the code tries to move the player only along the x axis, then only along the y axis, to find out which side of the tile the player collided with. If the player did not collide with any of the sides it hit a corner (that’s the real corner case!):

proc moveBox(map: Map, pos: var Point2d, vel: var Vector2d,
             size: Vector2d): set[Collision] {.discardable.} =
  let distance = vel.len
  let maximum = distance.int

  if distance < 0:
    return

  let fraction = 1.0 / float(maximum + 1)

  for i in 0 .. maximum:
    var newPos = pos + vel * fraction

    if map.testBox(newPos, size):
      var hit = false

      if map.testBox(point2d(pos.x, newPos.y), size):
        result.incl Collision.y
        newPos.y = pos.y
        vel.y = 0
        hit = true

      if map.testBox(point2d(newPos.x, pos.y), size):
        result.incl Collision.x
        newPos.x = pos.x
        vel.x = 0
        hit = true

      if not hit:
        result.incl Collision.corner
        newPos = pos
        vel = vector2d(0, 0)

    pos = newPos

The moveBox proc also returns a set of collisions, telling us what kind of collisions happened in this iteration. We don’t use that information, but it could be useful if you want to handle collisions in a special way instead of just pushing the player out of the collided wall.

Finally we can use onGround and moveBox:

proc physics(game: Game) =
  if game.inputs[Input.restart]:
    game.player.restartPlayer()

  let ground = game.map.onGround(game.player.pos, playerSize)

  if game.inputs[Input.jump]:
    if ground:
      game.player.vel.y = -21

  let direction = float(game.inputs[Input.right].int -
                        game.inputs[Input.left].int)

  game.player.vel.y += 0.75
  if ground:
    game.player.vel.x = 0.5 * game.player.vel.x + 4.0 * direction
  else:
    game.player.vel.x = 0.95 * game.player.vel.x + 2.0 * direction
  game.player.vel.x = clamp(game.player.vel.x, -8, 8)

  game.map.moveBox(game.player.pos, game.player.vel, playerSize)

Jumping is only possible when standing on the ground. Horizontal movement in the air is calculated in a different way than on the ground, simulating different air and ground friction.

Full code for section 5

6. Camera

Did you see me disappear out of the window? Nice, right?! Well, actually that probably makes playing the rest of the map rather difficult. So while we can move nicely, the camera’s position is still fixed. In this section we will only make the camera move horizontally. With this information you can probably figure out how to make it move vertically as well if you want to. We only need to set the camera position when our player moved, right after calling game.physics():

const
  windowSize: Point = (1280.cint, 720.cint)

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  game.camera.x = game.player.pos.x - halfWin

for tick in lastTick+1 .. newTick:
  game.physics()
  game.moveCamera()

Now the camera always immediately follows the player’s horizontal position. Another approach for the camera is to have it follow the player only once the player leaves the center area of the screen, in this case 200 pixels:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  let
    leftArea  = game.player.pos.x - halfWin - 100
    rightArea = game.player.pos.x - halfWin + 100
  game.camera.x = clamp(game.camera.x, leftArea, rightArea)

An alternative approach is to have the camera follow the player fluidly. When the player is further away from the center the camera moves towards him faster. You can imagine the camera as being pulled by a rubber band connecting it to the player, the further away they get, the stronger the camera gets pulled to the player:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  let dist = game.camera.x - game.player.pos.x + halfWin
  game.camera.x -= 0.05 * dist

Choices are difficult so we can just implement all three and you can choose at compile time using -d:fluidCamera and -d:innerCamera:

proc moveCamera(game: Game) =
  const halfWin = float(windowSize.x div 2)
  when defined(fluidCamera):
    let dist = game.camera.x - game.player.pos.x + halfWin
    game.camera.x -= 0.05 * dist
  elif defined(innerCamera):
    let
      leftArea  = game.player.pos.x - halfWin - 100
      rightArea = game.player.pos.x - halfWin + 100
    game.camera.x = clamp(game.camera.x, leftArea, rightArea)
  else:
    game.camera.x = game.player.pos.x - halfWin

Full code for section 6

7. Game State

Now that we can run around all of the map with the camera keeping up with us, we should give our game a purpose. You might have noticed the light and dark gray lines at the beginning and end of the map, suspiciously referred to as start and finish respectively. As you can probably guess we will use those as start and finish lines and record how quickly the player can get from start to finish:

type
  Time = ref object
    begin, finish, best: int

  Player = ref object
    texture: TexturePtr
    pos: Point2d
    vel: Vector2d
    time: Time

proc restartPlayer(player: Player) =
  player.pos = point2d(170, 500)
  player.vel = vector2d(0, 0)
  player.time.begin = -1
  player.time.finish = -1

proc newTime: Time =
  new result
  result.finish = -1
  result.best = -1

proc newPlayer(texture: TexturePtr): Player =
  new result
  result.texture = texture
  result.time = newTime()
  result.restartPlayer()

We’re now storing a Time object in the Player, telling us when the player began playing this round, how he finished last time and what his absolute best time is. By default the values are initialized to -1 to indicate an invalid value, otherwise they store ticks.

To format the time for display we use the excellent strfmt library’s string interpolation:

import strfmt

proc formatTime(ticks: int): string =
  let
    mins = (ticks div 50) div 60
    secs = (ticks div 50) mod 60
    cents = (ticks mod 50) * 2
  interp"${mins:02}:${secs:02}:${cents:02}"

Our game logic works as follows:

When the player walks through a start tile his time begins
When the player walks through a finish tile his finish time is set and printed to the terminal. If it is a new best time, so is his best time. The start time is reset.

proc getTile(map: Map, pos: Point2d): uint8 =
  map.getTile(pos.x.round.int, pos.y.round.int)

proc logic(game: Game, tick: int) =
  template time: untyped = game.player.time
  case game.map.getTile(game.player.pos)
  of start:
    time.begin = tick
  of finish:
    if time.begin >= 0:
      time.finish = tick - time.begin
      time.begin = -1
      if time.best < 0 or time.finish < time.best:
        time.best = time.finish
      echo "Finished in ", formatTime(time.finish)
  else: discard

We need to call the logic in our main proc of course:

for tick in lastTick+1 .. newTick:
  game.physics()
  game.moveCamera()
  game.logic(tick)

Now we can play through the map and finally get an output like this on the terminal:

Finished in 00:04:38

Full code for section 7

8. Text Rendering

It would be much nicer to have the text output with the result in the actual game window instead of on the terminal. For this we will now use SDL_ttf:

import sdl2/ttf

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y: cint, color: Color) =
  let surface = font.renderUtf8Solid(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  var source = rect(0, 0, surface.w, surface.h)
  var dest = rect(x, y, surface.w, surface.h)
  let texture = renderer.createTextureFromSurface(surface)

  sdlFailIf texture.isNil:
    "Could not create texture from rendered text"

  surface.freeSurface()

  renderer.copyEx(texture, source, dest, angle = 0.0, center = nil,
                  flip = SDL_FLIP_NONE)

  texture.destroy()

proc renderText(game: Game, text: string,
                x, y: cint, color: Color) =
  const outlineColor = color(0, 0, 0, 64)
  game.renderer.renderText(game.font, text, x, y, color)

What’s happening here is that we render a text with a FontPtr to an SDL2 surface (stored in RAM), which is then put into a texture (stored in GPU’s VRAM). This texture is then rendered to the screen at the defined position.

proc render(game: Game, tick: int) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)

  let time = game.player.time
  const white = color(255, 255, 255, 255)
  if time.begin >= 0:
    game.renderText(formatTime(tick - time.begin), 50, 100, white)
  elif time.finish >= 0:
    game.renderText("Finished in: " & formatTime(time.finish),
      50, 100, white)
  if time.best >= 0:
    game.renderText("Best time: " & formatTime(time.best),
      50, 150, white)

  # Show the result on screen
  game.renderer.present()

We need to initialize the TTF subsystem and now we also need to pass the current tick to render inside of our main function:

sdlFailIf(ttfInit() == SdlError): "SDL2 TTF initialization failed"
defer: ttfQuit()

...

game.render(lastTick)

As well as loading our DejaVuSans.ttf font inside of newGame:

result.font = openFont("DejaVuSans.ttf", 28)
sdlFailIf result.font.isNil: "Failed to load font"

The Game object now looks like this, including a font:

type
  Game = ref object
    inputs: array[Input, bool]
    renderer: RendererPtr
    font: FontPtr
    player: Player
    map: Map
    camera: Vector2d

This is what the text rendering looks like after finishing the map:

It noticeably doesn’t look all that great. The border of the text looks rugged. This happens because renderUtf8Solid does not use alpha blending, which is expensive. Instead every pixel is either entirely white or entirely transparent, never anything half-transparent in between. If we had a fixed background color for the text we could use renderUtf8Shaded, which takes a background color. If we want nicer output with dynamic backgrounds we can use renderUtf8Blended instead:

This looks better, but would be hard to see with a brighter background. We can draw an outline for our text to fix this, basically by drawing the text twice, once in half-transparent black and once in the proper color on top:

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y, outline: cint, color: Color) =
  font.setFontOutline(outline)
  let surface = font.renderUtf8Blended(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  var source = rect(0, 0, surface.w, surface.h)
  var dest = rect(x - outline, y - outline, surface.w, surface.h)
  ...

proc renderText(game: Game, text: string,
                x, y: cint, color: Color) =
  const outlineColor = color(0, 0, 0, 64)
  game.renderer.renderText(game.font, text, x, y, 2, outlineColor)
  game.renderer.renderText(game.font, text, x, y, 0, color)

Full code for section 8

9. Text Caching

If you looked at your CPU usage during this tutorial so far you might have noticed that the game needed nearly no CPU at all, about 3 % for 60 fps on my system. Once the texts are rendered this increases to 20% though. The reason for this is that right now we are regenerating the text textures every single frame, even if it didn’t change from the last frame. If you have fixed texts you can simply save the textures instead of recalculating them. But if you want more flexibility instead you can use a glyph or texture caching system. An example of such a system would be SDL_FontCache.

But instead we can write a little application-specific caching scheme in Nim. The heuristic we use is that mostly a single line in the code base will keep producing the same string, at least for some time. So we cache only a single text rendering for each line that prints something to the screen. That means we don’t have to do any lookups in a cache data structure and we only use a guaranteed constant amount of memory for caching:

type
  CacheLine = object
    texture: TexturePtr
    w, h: cint

  TextCache = ref object
    text: string
    cache: array[2, CacheLine]

proc newTextCache: TextCache =
  new result

A CacheLine is what we store, a pointer to a texture as well as the texture’s width and height. For our text rendering we need two of those as we render the text twice to get the outline effect. The text is also stored in TextCache to see if we already have the correct textures cached.

proc renderText(renderer: RendererPtr, font: FontPtr, text: string,
                x, y, outline: cint, color: Color): CacheLine =
  font.setFontOutline(outline)
  let surface = font.renderUtf8Blended(text.cstring, color)
  sdlFailIf surface.isNil: "Could not render text surface"

  discard surface.setSurfaceAlphaMod(color.a)

  result.w = surface.w
  result.h = surface.h
  result.texture = renderer.createTextureFromSurface(surface)
  sdlFailIf result.texture.isNil: "Could not create texture from rendered text"

  surface.freeSurface()

We manipulated renderText to return a CacheLine that we can use:

proc renderText(game: Game, text: string, x, y: cint,
                color: Color, tc: TextCache) =
  let passes = [(color: color(0, 0, 0, 64), outline: 2.cint),
                (color: color, outline: 0.cint)]

  if text != tc.text:
    for i in 0..1:
      tc.cache[i].texture.destroy()
      tc.cache[i] = game.renderer.renderText(
        game.font, text, x, y, passes[i].outline, passes[i].color)
    tc.text = text

  for i in 0..1:
    var source = rect(0, 0, tc.cache[i].w, tc.cache[i].h)
    var dest = rect(x - passes[i].outline, y - passes[i].outline,
                    tc.cache[i].w, tc.cache[i].h)
    game.renderer.copyEx(tc.cache[i].texture, source, dest,
                         angle = 0.0, center = nil)

In two passes the text is rendered, if the text is cached from the cache directly, otherwise the old cache entry is removed and replaced with the new textures.

Nim’s metaprogramming allows us to use this seamlessly with a small template. A few days ago I wrote an article about Metaprogramming in Nim if you want to learn more about the powerful side of Nim.

template renderTextCached(game: Game, text: string,
                          x, y: cint, color: Color) =
  block:
    var tc {.global.} = newTextCache()
    game.renderText(text, x, y, color, tc)

proc render(game: Game, tick: int) =
  # Draw over all drawings of the last frame with the default color
  game.renderer.clear()
  # Actual drawing here
  game.renderer.renderTee(game.player.texture,
    game.player.pos - game.camera)
  game.renderer.renderMap(game.map, game.camera)

  let time = game.player.time
  const white = color(255, 255, 255, 255)
  if time.begin >= 0:
    game.renderTextCached(formatTime(tick - time.begin),
      50, 100, white)
  elif time.finish >= 0:
    game.renderTextCached("Finished in: " &
      formatTimeExact(time.finish), 50, 100, white)
  if time.best >= 0:
    game.renderTextCached("Best time: " &
      formatTimeExact(time.best), 50, 150, white)

  # Show the result on screen
  game.renderer.present()

Now each of the three renderTextCached calls get its own TextCache assigned, which is then used for the rest of the program execution. Note that this caching scheme only works under the assumption that there are relatively few separate lines of code that call renderTextCached and the ones that do often render the same text multiple times in a row. Good enough for our use case.

proc formatTime(ticks: int): string =
  let mins = (ticks div 50) div 60
  let secs = (ticks div 50) mod 60
  interp"${mins:02}:${secs:02}"

proc formatTimeExact(ticks: int): string =
  let cents = (ticks mod 50) * 2
  interp"${formatTime(ticks)}:${cents:02}"

We also reduced the exactness of the current time format because before it had to be recalculated every single frame. Now we’re back to using 4 % CPU and our final game looks like this:

Full code for section 9

Building

We can create a platformer.nimble file to tell Nimble, Nim’s package manager, how to build our package:

# Package

version       = "1.0"
author        = "Dennis Felsing"
description   = "An example platform game with SDL2"
license       = "MIT"

bin           = @["platformer"]

# Dependencies

requires "nim >= 0.10.0"
requires "sdl2 >= 1.1"
requires "strfmt >= 0.6"
requires "basic2d >= 0.1.0"

task tests, "Compile all tutorial steps":
  for i in 1..9:
    exec "nim c tutorial/platformer_part" & $i

All the intermediate code states from this article can be compiled with nimble tests. We can test the build by running nimble build, which creates a platformer binary that we can run. nimble install installs the same binary so that it is available in ~/.nimble/bin.

But once we run this installed platformer binary we get an error:

Error: unhandled exception: Could not load image player.png, SDL error: Couldn't open player.png [SDLException]

Actually this makes sense, because we load the files at runtime from the current directory, which can now be any directory. We have two choices for how to solve this:

Read in all files at compile time and get a binary that depends on no assets
Doing this is pretty simple in Nim as the compiler supports reading arbitrary files at compile time with staticRead:

import streams

template staticReadRW(filename: string): ptr RWops =
  const file = staticRead(filename)
  rwFromConstMem(file.cstring, file.len)

template staticReadStream(filename: string): string =
  const file = staticRead(filename)
  newStringStream(file)

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

  result.font = openFontRW(
    staticReadRW("DejaVuSans.ttf"), freesrc = 1, 28)
  sdlFailIf result.font.isNil: "Failed to load font"

  result.player = newPlayer(renderer.loadTexture_RW(
    staticReadRW("player.png"), freesrc = 1))
  result.map = newMap(renderer.loadTexture_RW(
    staticReadRW("grass.png"), freesrc = 1),
    staticReadStream("default.map"))

Define a directory where we store our data assets to load them at runtime. This enables the player to switch them out for custom ones without the need to recompile the binary. And as we’ve seen with the DDNet Skin Database this might be a useful feature. We find the data directory by looking where in the file system our binary is. Let’s implement this and make the old embedded compile time assets optional:

import os, streams

const dataDir = "data"

when defined(embedData):
  template readRW(filename: string): ptr RWops =
    const file = staticRead(dataDir / filename)
    rwFromConstMem(file.cstring, file.len)

  template readStream(filename: string): Stream =
    const file = staticRead(dataDir / filename)
    newStringStream(file)
else:
  let fullDataDir = getAppDir() / dataDir

  template readRW(filename: string): ptr RWops =
    var rw = rwFromFile(cstring(fullDataDir / filename), "r")
    sdlFailIf rw.isNil: "Cannot create RWops from file"
    rw

  template readStream(filename: string): Stream =
    var stream = newFileStream(fullDataDir / filename)
    if stream.isNil: raise ValueError.newException(
      "Cannot open file stream:" & fullDataDir / filename)
    stream

proc newGame(renderer: RendererPtr): Game =
  new result
  result.renderer = renderer

  result.font = openFontRW(
    readRW("DejaVuSans.ttf"), freesrc = 1, 28)
  sdlFailIf result.font.isNil: "Failed to load font"

  result.player = newPlayer(renderer.loadTexture_RW(
    readRW("player.png"), freesrc = 1))
  result.map = newMap(renderer.loadTexture_RW(
    readRW("grass.png"), freesrc = 1),
    readStream("default.map"))

This also requires us to change newMap to accept a Stream instead of a filename:

proc newMap(texture: TexturePtr, map: Stream): Map =
  new result
  result.texture = texture
  result.tiles = @[]

  var line = ""
  while map.readLine(line):
    ...

When compiling you can set -d:embedData like this:

$ nim -d:release c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 129K Jun 13 14:54 platformer*
$ nim -d:release -d:embedData c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 888K Jun 13 14:55 platformer*

You can find our final platform game code in the repository.

Now we can submit the repository to the Nimble packages as a pull request and soon all Nim developers can install the package right from their terminal with a nimble install platformer and play by just running platformer.

A circle.yml file defines how to run and compile our repository and is used to make sure it keeps building fine whenever changes are made.

Binary Distribution

But Nim developers are probably not the main target group of our game, so ideally we we also want to be able to build binaries for a few common platforms, Linux x86, x86-64 and Windows x86, x86-64 for us. Building for Mac OS X is a bit more involved, but you can check out how DDNet does it.

Of course we could just set up a VM for each system that we want to build for and use the instructions from the start of this article. But that’s tedious and we want the convenience of building on a single machine.

Linux

Note that I’m on Arch Linux, but this should be possible in an analogous way on other Linux distributions:

Building a portable binary for Linux is a pain because of glibc. When you compile on a system with a newer glibc version it might not run on a system with an older one. A common solution is to use a build system with an old Linux install. Alternatively an old Debian chroot can be created with debootstrap. There is also Linux Standard Base which aims to solve this problem, but I haven’t used yet.

A more hacky solution is to create the binary on the new system and check what symbols exactly are linked against the newer glibc version. In our case I want everything to use GLIBC_2.2.5, so I check for anything else:

objdump -T platformer | grep GLIBC | grep -v 2.2.5
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy

Okay, only memcpy is the problem. We can force the linker to use an old version of memcpy and another common problem, realpath, like this in C code with inline assembler:

__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");

But this would require for us to insert this into every C file that we generate. Or we abuse the nimbase.h file and insert it there and compile with:

$ head -n2 glibc-hack/nimbase.h
__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");
$ nim -d:release --passC:-Iglibc-hack c platformer
$ objdump -T platformer | grep memcpy
0000000000000000      DF *UND* 0000000000000000  GLIBC_2.2.5 memcpy
$ objdump -T platformer|grep GLIBC|grep -v 2.2.5

Now our binary works on Linux versions using glibc 2.2.5 or newer. Note that the user still needs SDL2, SDL_image2 and SDL_ttf2 installed.

When you’re linking dynamically and want to distribute shared libraries with your binary you can compile with nim --passC:-Wl,-rpath,. c platformer and put the shared libraries into the same directory as the binary.

At least building for x86 is easy on Linux as long as you’re on x86-64 and have gcc-multilib installed:

$ yaourt -S lib32-sdl2 lib32-sdl2_image lib32-sdl2_ttf
$ nim --cpu:i386 --passC:-m32 --passL:-m32 -d:release c platformer

Windows

Surprisingly it is much easier to build portable binaries for Windows, even from Linux:

$ pacman -S mingw-w64-gcc
$ nim --os:windows --cpu:amd64 --gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc -d:release c platformer
$ nim --os:windows --cpu:i386 --gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc -d:release c platformer

SDL libraries for Windows can be downloaded from the SDL2 website (image, ttf).

Stripping the binaries of symbols with strip -s platformer is a good idea if you want to save some space and don’t care about debugging your binary.

Automated Build Script

With all this information we can now write a fully automated release build script, written in Nim as well:

import os, strfmt

const
  app = "platformer"
  version = "1.0"

  builds = [
    (name: "linux_x86", os: "linux", cpu: "i386",
     args: "--passC:-m32 --passL:-m32"),
    (name: "linux_x86_64", os: "linux", cpu: "amd64",
     args: "--passC:-Iglibc-hack"),
    (name: "win32", os: "windows", cpu: "i386",
     args: "--gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc"),
    (name: "win64", os: "windows", cpu: "amd64",
     args: "--gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc"),
  ]

removeDir "builds"

for name, os, cpu, args in builds.items:
  let
    dirName = app & "_" & version & "_" & name
    dir = "builds" / dirName
    exeExt = if os == "windows": ".exe" else: ""
    bin = dir / app & exeExt

  createDir dir
  if execShellCmd(interp"nim --cpu:${cpu} --os:${os} ${args} -d:release -o:${bin} c ${app}") != 0: quit 1
  if execShellCmd(interp"strip -s ${bin}") != 0: quit 1
  copyDir("data", dir / "data")
  if os == "windows": copyDir("libs" / name, dir)
  setCurrentDir "builds"
  if os == "windows":
    if execShellCmd(interp"zip -9r ${dirName}.zip ${dirName}") != 0: quit 1
  else:
    if execShellCmd(interp"tar cfz ${dirName}.tar.gz ${dirName}") != 0: quit 1
  setCurrentDir ".."

Linux users have to install sdl2, sdl2_image, sdl2_ttf using their package manager. Windows users get them bundled. Our build script creates this directory structure when we run it with nim -r c release:

.
├── platformer_1.0_linux_x86
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   └── platformer
├── platformer_1.0_linux_x86.tar.gz
├── platformer_1.0_linux_x86_64
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   └── platformer
├── platformer_1.0_linux_x86_64.tar.gz
├── platformer_1.0_win32
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   ├── libfreetype-6.dll
│   ├── libpng16-16.dll
│   ├── platformer.exe
│   ├── SDL2.dll
│   ├── SDL2_image.dll
│   ├── SDL2_ttf.dll
│   └── zlib1.dll
├── platformer_1.0_win32.zip
├── platformer_1.0_win64
│   ├── data
│   │   ├── default.map
│   │   ├── DejaVuSans.ttf
│   │   ├── grass.png
│   │   └── player.png
│   ├── libfreetype-6.dll
│   ├── libpng16-16.dll
│   ├── platformer.exe
│   ├── SDL2.dll
│   ├── SDL2_image.dll
│   ├── SDL2_ttf.dll
│   └── zlib1.dll
└── platformer_1.0_win64.zip

Resulting downloads here: Win64, Win32, Linux x86_64, Linux x86

Final Words

Somehow my articles are getting longer and this journey towards writing a simple platform game took a few more explanations than expected. I hope this article wasn’t too long and was instructive and helpful to get an understanding of how to write platform games in Nim with SDL2.

All the material for this article is available in the repository on GitHub. Because I changed around things late into the article I might have made a mistake or missed something. If you find a bug or have a comment you can drop me an email at dennis@felsing.org.

Discussions on r/programming and Hacker News.

Experiences of Running an Online Game for 3 Years

2016-06-09T00:00:00-04:00

It’s been roughly 3 years since DDraceNetwork (DDNet) started in the summer of 2013. Last year I wrote a non-technical History of DDNet. Today in this post we will dive into the technical side of what makes DDNet run.

For the uninitiated, DDNet is an open-source modification of Teeworlds, a retro multiplayer shooter. DDNet is a game in which you, instead of killing each other, cooperate with each other to work your way through challenging maps, trying to beat the map or get a better time than other teams.

What we offer for DDNet is the client and server software as well as official servers that run all around the world. We have an international community of players trying to beat the latest maps and records, and dozens of mappers who send in their newest creations for our consideration. A new map is released every few days and occasionally tournaments happen in which the best compete against each other on brand-new maps.

At the time of writing there are 968 people playing Teeworlds, 678 of which are on a server running the DDNet mod, and 430 on the official DDNet servers. So this project is not especially big and not many people stumble upon DDNet. Our player base consists in a large part of old Teeworlds players and their friends who heard about us by word of mouth.

Nevertheless I hope that the technical challenges we faced and face as well as our solutions are interesting. Let’s start with an overview of what we have:

Overview & Financing

DDNet runs official servers in Germany, Russia, USA, Canada, Chile, Brazil, China, South Africa and Iran.

Since DDNet is totally free, has no advertisements and no in-game purchases, it offers no stream of revenue and is solely funded by donations, donated servers and my own money. So one of the goals is to keep costs down: On average we pay 10 € per month for each location.

Let’s see what we can offer with this to our thousands of players (detailed statistics) and how DDNet evolved to keep with increasing number of players, maps and the product of both, ranks.

Servers & Locations

We exclusively use cheap virtual private servers (VPS). They offer enough performance for us and are significantly cheaper than dedicated servers. To have a bit stronger guarantees on performance I prefer KVM and XEN over OpenVZ, but there are good and bad hosters for each. In the end you always have to try out a hoster for a few days up to a month to find out if it is suitable for hosting official DDNet servers. The most important criteria for us are:

Good and consistent CPU performance: should do about 400 players per CPU core
Low network latency to our players in the region: DDNet is only enjoyable with a low and stable ping
0.5 to 1 GB of RAM is enough for ~~everyone~~ most locations: running about 20 game servers per location

When searching for hosters in a country you should prefer to look in the language of the country instead of English. These tend to be cheaper and less overcrowded by international buyers. Be prepared to use a translator and have trouble with the support, but after some time all these hoster websites start to look the same to you, no matter if they’re in English, German, French, Spanish or Farsi.

When in doubt, ask locals what hosters they use or recommend, or check the whois entries for servers in the region that seem to run well. For example the whois 31.186.251.128 entry tells you that DDNet.tw (and this blog) are hosted at Nuclear Fallout Enterprise, connected by the InterNAP Network:

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '31.186.250.0 - 31.186.251.255'

% Abuse contact for '31.186.250.0 - 31.186.251.255' is 'noc@internap.com'

inetnum:        31.186.250.0 - 31.186.251.255
netname:        INAP-LON-nuclearfallout-31-186-250-0
descr:          Nuclear Fallout Enterprise
country:        DE
admin-c:        INAP-RIPE
tech-c:         INAP-RIPE
status:         ASSIGNED PA
mnt-by:         INAP-MAINT-RIPE
created:        2013-08-08T19:09:47Z
last-modified:  2014-03-06T23:07:27Z
source:         RIPE # Filtered

person:         InterNAP Network Operator
address:        InterNAP Network Services
address:        Two Union Square
address:        601 Union Street Suite 1000
address:        Seattle, WA 98101 USA
phone:          +1 206 441 8800
fax-no:         +1 206 256 9580
nic-hdl:        INAP-RIPE
mnt-by:         INAP-MAINT-RIPE
created:        2002-03-04T12:57:06Z
last-modified:  2002-03-04T12:57:06Z
source:         RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.87.3 (DB-2)

Unfortunately the story doesn’t end here for us. Since DDNet is a multiplayer open-source game we seem to attract the kind of people who know how to find a DDoS (distributed denial of service) booter. This resulted in us being kicked out at many hosters since they either do not have any (D)DoS protection or it is not sufficient against some attacks, which then also affect other customers of theirs.

In the end we settled on NFOservers for our main servers, While they do not advertise any DDoS protection, they turned out to be the only one of dozens of hosters I tried that can withstand most of the attacks we get regularly. I’m of course talking about cheap VPSes, I hear that good DDoS protection exists for a few hundred euros per month, which is far outside of our reach. Unfortunately NFOservers is only available in USA and Germany.

Here’s what the attacks in a regular 2 week period looks like in our most popular location, Europe (GER server):

In Russia (Moscow) we had ok experiences with reg.com, at least the ping for local players is lower than in any other location.

Recently we got a server hosted in Iran again, which has always been a difficult country to host a server in, for matters of cost as well as their preference of an intranet over the internet.

In Chile we’re now with zGlobalHost, one of the few hosters with unlimited bandwidth and by far the best I found for just 9 € per month. Their server room looks very cozy:

In Chile we also had this bizarre story back in January, after the entire data center of ZGlobalHost went down:

January 26, 16:04 (local time Chile)
Our provider GTD has trouble again and is still not giving estimated time of repair. We will inform you with more news as soon as possible.
January 26, 18:06
Our provider informs us that in the next 30 minutes everything should be operational again.
January 26, 19:09
GTD reports that there are two fiber cuts, one in Pudahuel and another in the Eastern Zone. They are certifying the fibers and lifting services one by one.
January 26, 22:34
Be informed that when we are back online and GTD reports that it will provide further details about the incident as it was just reported that the fiber cut was due to vandalism, the streets in Providencia are now reported to be safe for repair works again.
We do not have backup lines. It will at least take another hour but repairs should not take longer than until this morning. You will be mailed a report instantly when the servers are back online and tomorrow GTD will send a report with details.
January 26, 22:36
The new details we have say that there were 4 cuts today, which is why there was such a long delay:
Locations: Pudahuel, Providencia, La Florida, Macul
Causes for each cut: Fire, Vandalism, Highway Accident, Material fatigue in electric pole cut it down
All ISPs in various areas have been affected, particularly fiber optic, both business and home addresses.
January 27, 03:00
Service is fully operational again since 02:30 AM.

To make sure that our servers keep running well, we monitor them and record the server statistics. When an unexpected event happens, like a server downtime or high network traffic, a notification is automatically sent. You can read more about this system in a separate post from last month.

Software Platform

As the operating system our servers run Linux, preferably Debian, but also some instances of Ubuntu and CentOS. In areas like South Africa, South America and Iran bandwidth can be exorbitantly expensive, so when you get a donated server in one of those locations you can not afford to be picky.

For the databases we use MariaDB, the drop-in MySQL fork started by the original MySQL developers after Oracle acquired it. We use this database to record our maps, saved games and the ranks of all players worldwide and keep these synchronized around the world.

Initially we started with a single VPS and a database running directly on it. As we grew to add more locations around the world we used a circular master-slave replication. With time some servers turned out to be less stable than others, so they were taken out of the loop, their game servers communicating with nearby MariaDB servers instead:

Now we’re switching away from this circular master-slave replication to having a small number of master databases which are replicated by slaves at each location. This has the advantage that database reads happen locally, so getting your own rank is always fast. Meanwhile new ranks are sent to a master server when possible or otherwise stored in a local file to be added later. This is necessary since our cheap servers are not always connected to the entire internet, sometimes they are only available from inside their own country.

The DDNet client and server, being based on Teeworlds, are also written in C/C++ (also known as C with classes). Luckily their performance was good from the start, but a few optimizations still helped with keeping the load low when running up to 60 game servers on a single VPS, for example:

Reduce the number of syscalls by caching the value of gettimeofday() until a new tick happens or network packet comes in
Do not execute the main game loop when the server is currently empty, as nothing will happen with no players anyway
Use CPU (nice) and IO (ionice) priorities to prevent background tasks (lower priority) to get in the way of game server processes (high priority)
Each VPS compiles its own server binary tuned for the specific target architecture
Prefer to download map files over HTTP from our web server instead of the built-in UDP protocol that can be shaky and slow. This also reduces the network traffic on game servers, keeping network jitter lower.
Instead of downloading the entire archive on a new client release, update only the changed files

Generally the idea is to wait until you notice that some resource is being used too much, benchmark and analyze what the reason for it is, and only then optimize this.

Website

Back when DDNet started in 2013 the website ran on the same VPS as our game servers. Later we switched to hosting the website on a separate VPS, in part to improve the DDoS protection by having mostly UDP on game servers and mostly TCP on the web server, but also to make sure that the website traffic and CPU usage do not impact the game servers.

This has become even more important since the map downloads are now also happening over HTTP from the web server instead of going over UDP from the game servers. Automatically updating the list of available maps is done with a small Nim script that I already presented in a previous article:

#!/usr/bin/env nimscript

import os, crc32, strutils

const
  baseDir = "/home/teeworlds/servers"
  mapdlDir = "/var/www-maps"

for kind, path in walkDir baseDir/"maps":
  if kind != pcFile:
    continue

  let (dir, name, ext) = splitFile(path)

  if ext != ".map":
    continue

  let
    sum = crc32FromFile(path).int64.toHex(8).toLower
    newName = name & "_" & sum & ext
    newPath = mapdlDir / newName
    tmpPath = newPath & ".tmp"

  if existsFile newPath:
    continue

  copyFile path, tmpPath
  moveFile tmpPath, newPath

Most of the DDNet.tw website is statically built by jekyll and automatically deploying using GitHub’s webhooks. That’s a simple solution and requires very few resources.

Pages about player status, ranks, map releases and mappers are also statically built, but by Python scripts directly on the server. This also makes sense since these pages are expensive to build and are requested more often than they have to be generated.

We also used to generate the player pages in the same way, but since we have about 90,000 ranked players so far this would’ve become a bit too expensive. Instead we run a small uWSGI Python server to dynamically generate them now from the data kept in memory. Right now it takes about 40-80 ms to generate a single player page, by far fast enough for us. If the computation time goes up or we suddenly become much more popular caching could be implemented.

We used to package a growing selection of skins with the DDNet client, but maintaining this became too cumbersome and increased memory usage for the client. Now instead the website offers a database of skins where you can select individual skins or skin packs.

Software Releases

DDNet has been in active development for the last 3 years, with our first client & server software release back in October 9, 2013.

Initially I used to build the releases manually using virtual machines for Windows and Linux, later also Mac OS X. After some time this got tedious of course, so I wrote an automated build script. It uses a Debian chroot for Linux building, MinGW for Windows and a QEMU VM for Mac OS X. I set up the Mac OS X QEMU image using a continuously updated guide and it works just fine. I connect to the guest OS through a simple ssh connection that is forwarded to the virtual machine:

# Start the Mac OS X VM
qemu-system-x86_64 [...] \
  -netdev user,id=hub0port0,hostfwd=tcp::10022-:22 \
  -device e1000-82545em,netdev=hub0port0,id=mac_vnet0 \
  &>/dev/null &

# Wait until VM is booted up
while ! ssh -p 10022 -o ConnectTimeout=10 localhost exit; do true; done

# Run build script
ssh -p 10022 localhost "# Run build script for Mac OS X"

Building for Android is also possible but disabled right now because we still have to port the Android version to SDL2, and DDNet is not really playable on Android anyway since a mouse is the most important tool in the game.

When you think about it, it’s pretty amazing that you can easily cross-compile on a single machine for all of these targets:

Windows x86, x86_64
Linux x86, x86_64
Mac OS X
Android armeabi, armeabi-v7a, mips, x86

The automated builds happen on my cheap ASRock Q1900-ITX home sever (which was also used to stream gameplay from DDNet servers) and still only take a few minutes to complete:

Finally when all goes well you get a small summary of the build times that looks like this:

Preparation:      10 s
Linux x86_64:     62 s
Linux x86:        64 s
Windows x86_64:  106 s
Windows x86:      84 s

For the automated builds we use bundled static libraries, while people who build our software locally probably prefer to use their system libraries. Using pkg-config we detect whether libraries are available at compile-time and otherwise use the static ones if they are available. This behaviour can be controlled manually as well:

# Analog to ./configure
bam config curl.use_pkgconfig=false \
           opus.use_pkgconfig=false \
           opusfile.use_pkgconfig=false \
           ogg.use_pkgconfig=false
# Analog to make
bam release

Even a JavaScript port of DDNet client is available for playing without any installation, compiled with Emscripten. For this purpose the official servers also accept WebSocket connections additionally to UDP.

New releases are added to the website with a nice changelog. But most existing players get their update notification in the client directly and run the updater from there, only downloading the files that actually changed instead of the entire new release.

Map Testing & Releases

The best part about DDNet is its active community. New maps for our servers are regularly sent in to be tested and later released. Initially the testing process was implemented as Trac installation. But it turned out that the non-technical people did not appreciate it much, so we switched to a regular phpBB forum for testing and communication.

New maps can be uploaded by our testers to the test servers and then tested there. The map upload happens with an upload script to the web server. The test servers around the world use sshfs to access these maps.

Once a new map has been cleared by the testers and all bugs fixed by the mapper, it can be released on the proper server. The preparation for this happens on the web server and is then synchronized to the other servers using an internal git repository.

The new map is also announced on the recent map releases page as well as its feed. Players on the official servers are informed about the map release using a broadcast message.

It’s possible to take a look at our maps using a WebGL renderer that supports parts of the map format, but no animations for example: Lonely map

When a popular new map is released, it’s possible to have hundreds of players taking up the challenge of playing the new map at once, either in small teams or in big groups.

Finally every day the newly released maps are added to our public maps repository, which can be used to easily run a clone of the official DDNet servers at home or on your own server. Just download the repository, add the DDNet-Server binary to the same directory and run it. A simple shell script suffices to update this repository:

#!/usr/bin/env zsh
mkdir -p /home/teeworlds/ddnet-maps
cd /home/teeworlds/ddnet-maps
rm -rf types

echo 'add_path $USERDIR\nadd_path $CURRENTDIR' > storage.cfg

for i in `cat ../servers/all-types`; do 
  TYPEDIR=types/${i:l}
  echo "add_path $TYPEDIR" >> storage.cfg
  mkdir -p $TYPEDIR/maps
  grep "|" ../servers/$TYPEDIR/maps | cut -d"|" -f2 | while read j; do 
    cp -- "../servers/maps/$j.map" $TYPEDIR/maps
    cp ../servers/$TYPEDIR/flexvotes.cfg $TYPEDIR
    grep -v "flexname.cfg" ../servers/$TYPEDIR/flexreset.cfg > $TYPEDIR/flexreset.cfg
    tail -n +5 ../servers/$TYPEDIR/votes.cfg > $TYPEDIR/votes.cfg
  done
done

git add * &>/dev/null
git commit -a -m "daily update" &>/dev/null
git push &>/dev/null

Server Types

In the start DDNet hosted only new maps categorized by their difficulty:

Later we also added categories for maps from old servers:

Oldschool: Ancient maps
DDmaX: Old DDRace server before DDNet

As well as categories for different kinds of single player maps:

Solo
Race: Classic mod with fewer features, converted maps to new DDNet format
Dummy: Maps where you play with another player that is unable to move

The idea is that, since we are the major hoster in many countries, many of these good maps would be forgotten forever if we don’t host them. At the same time we try to motivate people to make creative new maps, having worked together with mappers to add new features that they require.

While the DDNet servers are the main focus, we also use our servers for hosting a few other Teeworlds mods since they use very few resources and the servers are running already anyway.

Tournaments

When a well-known mapper makes a special map, they can choose to have their map played at an official DDNet tournament. For this it is necessary to keep the map under wraps, showing it only to a few select testers. This ensures that none of the players at the tournament will be familiar with the map already, which would give them an unfair advantage.

Usually tournaments are done on Sunday, 20:00 Central European Time since most of our player base is located in Europe. You can read about the work that goes into preparing a tournament. But the really stressful part is actually holding the tournament.

Since it runs at the same time on all (currently 9-10) servers worldwide, you need to control them all at once. If the tournament map has not been tested well enough it might need to be quickly fixed and reloaded during the tournament. Sometimes problems occur on a single server. Unfortunately tournaments also attract DDoS attacks.

Here you can see the last tournament’s top run, for which the players had 2 hours to compete:

Some of our mappers even created fancy ingame loading screens for the start of the tournament:

Client & Server Development

Development of the client and server software is often challenging because of backwards compatibility:

People with regular Teeworlds client should still be able to play on DDNet
People with old DDNet client (or custom client) should still be able to play
Old ranks would be invalidated by slight fixes/changes in game physics

Nevertheless we managed to put in many new features, for example:

64 player servers instead of just 16 players:
Players with vanilla Teeworlds client only see the 15 closest players on a server.
Protect players from connection timeouts:
The timeout protection works by the client having a unique timeout code that it sends to the server whenever it connects. This tells the server if the player has had a timeout before and they can get back their ingame character.
Allow saving game in team and continuing at a later time:
The save games are stored in our MariaDB database, but can only be loaded at a single location. Otherwise players would have the ability to load a save game multiple times, which is not what we want.
Save the ranks of teams and not just single players
SDL2 client instead of old SDL1.2
New physics: Jetpack, extended teleporters, walljump, tune zones
Highly compressed Opus sounds in maps, it’s amazing how much quality you get with small bit rates
In-client updater using HTTPS and a simple JSON file for control
Dummy feature to control two ingame characters at once:
At first this was meant to help map testers to be able to test maps easier, without requiring a second player. But it’s also very popular with regular players as well as mappers who come up with interesting maps where only one player has all the control.
…

Particularly important for the automation of the official DDNet servers is our FIFO console system. Every server listens for commands from a FIFO file. This enables us to automate the map releases, server restarts and broadcasts among other things. To broadcast a message about a new map release to all servers, you can just do this:

echo 'broadcast "New map Uniswim by Im corneum just released on Solo server"' > servers/*.fifo

We have a small Python script that can do this automatically and also broadcast in a custom big ASCII art font:

Using the FIFO system at multiple locations at once is achieved using Cluster SSH.

We also offer a list of official DDNet servers in JSON which is used to inform the client of the official servers as well as internally to generate the status page.

Unfortunately attacks with spoofed IP addresses are quite common for us. So we have added a spoofing protection using tokens to prevent attackers from filling up servers with players with spoofed IPs.

We have no automated tests for the client and server, the community quickly catches new bugs though. At least we use automated Circle CI building to make sure new changes still compile fine.

Communication

Ingame communication is a bit tricky since the game is built on simplicity and we prefer not to have accounts. Introducing accounts this late into existance would give another vector of abuse by giving attackers the chance to register other people’s names. So you can never be sure who you’re actually talking to. Thanks to some Social Engineering this has been a common source of leaked moderator passwords.

So for official communications the DDNet forum is used instead. We tried to use Let’s Encrypt for SSL/TLS certificates for the website, but it turned out that Windows XP is not supported and many of our players have old systems still running Windows XP. I guess that’s in the nature of running one of the few actively developed online games that work perfectly fine on 15 year old machines. Instead we’re now using StartSSL, which seems to work fine, except that wildcards would cost money. Once Let’s Encrypt works fine on Windows XP we might be able to switch back.

Otherwise technically inclined people mostly discuss on IRC (#ddnet on QuakeNet, WebChat, Logs) and Skype. Most non-technical discussions happen via Skype.

Future Directions

The main goal is to keep DDNet running, well maintained, and keep a steady stream of new maps from the community.

We submitted DDNet to Steam Greenlight and have been greenlit since then, which means we could release it on Steam now. This would be an opportunity for us to reach more potential players, especially those who have never played Teeworlds before. The main problem right now is that DDNet is a difficult game to get started with, since it requires practice, fine control and patience. Right now work is being done to create a tutorial for new players and generally improve the ease of getting started.

Discussion

You can find discussions about this article on Hacker News, r/programming and also the DDNet Forum.

DDNet Server Statistics with ServerStatus, RRDtool and Nim

2016-05-14T00:00:00-04:00

About a month ago I set up statistics for the official DDNet servers. My motivations for this are:

Monitor the servers more easily
Get notified about server problems
Have nice graphs to look at

The choices for the software used are mainly made to keep resource usage low, a general principle used for DDNet since we run on cheap VPSes all around the world and are limited in CPU and memory resources. In the rest of this post we will explore the 3 major tools used, their purpose in our solution as well as their performance impact:

ServerStatus: Gather live server statistics
RRDtool: Record and graph data
Nim: Favorite programming language for performance and readability

Gathering live server statistics with ServerStatus

We’ve been running BotoX’s ServerStatus to get live server statistics for some time now. It works quite well to quickly notice major server problems like a high load or incoming (D)DoS attacks, provided that you keep an eye out for it.

We use ServerStatus by running its simple Python client on each server to gather interesting information. The client transmits that data by TCP to the C/C++ server, which aggregates it into a JSON file. This JSON file is then fetched and displayed every two seconds by the JavaScript frontend of our Status page.

On a regular Saturday morning the end result looks like this:

On a quick glance you notice that the DDNet.tw server has high CPU usage (which is totally normal since it runs some hefty cron jobs every 20 minutes) and DDNet RUS is receiving a small DoS attack with just 1.6 MB/s (unfortunately also totally normal). Apart from that everything looks fine.

ServerStatus footprint, calculated from Linux /proc statistics as follows (in Nim):

import os, strutils, posix, strfmt
# CPU and memory usage of process, based on PROC(5) and
# http://stackoverflow.com/a/16736599
let
  pid = paramStr(1)
  # Clock ticks per second
  frequency = sysconf(SC_CLK_TCK)
  # Size of memory page in bytes
  pagesize  = sysconf(SC_PAGESIZE)

  uptime    = readFile("/proc/uptime").split[0].parseFloat
  fields    = readFile("/proc/" & pid & "/stat").split

  # Amount of time in user mode
  utime     = fields[13].parseInt
  # Amount of time in kernel mode
  stime     = fields[14].parseInt
  # Amount of children time in user mode
  cutime    = fields[15].parseInt
  # Amount of children time in kernel mode
  cstime    = fields[16].parseInt
  # Time process started after boot
  starttime = fields[21].parseInt

  # Resident Set Size: number of pages in memory
  rssmem    = fields[23].parseInt

  totaltime = utime + stime + cutime + cstime
  seconds   = uptime - (starttime / frequency)

  cpuusage  = 100 * (totaltime / frequency) / seconds
  memusage  = rssmem * pagesize / 1_000_000

echo interp"${cpuusage:.2f} % CPU ${memusage:.2f} MB Memory"

Part	CPU	Memory
Client	0.14 %	3.70 MB
Server (9 clients)	0.58 %	0.18 MB

Recording and graphing data with RRDtool

I haven’t used RRDtool for about 7 years, but it’s still an excellent tool to record data into a fixed-size round robin database. For us three functions of RRDtool are important: create to create the database, update to add a new value to be aggregated into the database, and graph to render the database into a beautiful graph.

CPU, network and memory are the most important resources for me, so their usage should be recorded. Let us use network traffic as an example and create a database:

First we need to think about what data we want to record in the RRD:

sample = 30 seconds

day  =   2880 samples =   6 * 480 pixels, each pixel is 03:00 min
days =  20160 samples =  42 * 480 pixels, each pixel is 21:00 min
days = 141120 samples = 147 * 960 pixels, each pixel is 73:30 min

Then we can use this to create the actual database file:

rrdtool create ddnet.tw-net.rrd `# File name` \
  --step 30 `# Interval in seconds with which data is fed` \
  DS:network_rx:GAUGE:60:0:U `# Data source receiving` \
  DS:network_tx:GAUGE:60:0:U `# DS Sending` \
  RRA:AVERAGE:0.5:6:480 `# Round robin archive for 1 day` \
  RRA:AVERAGE:0.5:42:480 `# RRA for 7 days` \
  RRA:AVERAGE:0.5:147:960 `# RRA for 49 days`

If you’re curious about what exactly happens here, you can find more information in rrdcreate(1).

The resulting ddnet.tw-net.rrd file is just 32 KB in size and will forever stay that exact size. (All our databases together are just 1 MB.) New data in each round robin archive simply overwrites the oldest data. A disadvantage of RRDtool is that you need to think ahead and plan what data you want to store.

The next step is to put new data into our little database, which we should do every 30 seconds:

rrdtool update ddnet.tw-net.rrd N:42:1234

Super simple! 42 is our network_rx value, 1234 the value for network_tx. These values are now aggregated using the AVERAGE and finally put into their respective archives.

Once we have enough values we can finally create the graph, for example for 1 day:

rrdtool graph ddnet.tw-net-1d.png --rigid --base 1000 \
  --width 419 --height 150 --logarithmic --units=si -a PNG \
  `# Calculation over last day only` \
  --vertical-label "Bytes/s" --start now-1d \
  `# Fetch data from RRD file` \
  DEF:network_rx=ddnet.tw-net.rrd:network_rx:AVERAGE \
  DEF:network_tx=ddnet.tw-net.rrd:network_tx:AVERAGE \
  `# Calculate aggregates based on data` \
  VDEF:network_rx_a=network_rx,AVERAGE \
  VDEF:network_rx_m=network_rx,MAXIMUM \
  VDEF:network_rx_c=network_rx,LAST \
  VDEF:network_rx_s=network_rx,TOTAL \
  VDEF:network_tx_a=network_tx,AVERAGE \
  VDEF:network_tx_m=network_tx,MAXIMUM \
  VDEF:network_tx_c=network_tx,LAST \
  VDEF:network_tx_s=network_tx,TOTAL \
  `# Draw area graph in light colors` \
  AREA:network_tx#fee8c8: \
  AREA:network_rx#e0e0e0: \
  `# Draw clear area outline on top` \
  LINE1:network_tx#e34a33:"out" \
  `# Print aggregate values to legend` \
  GPRINT:network_tx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_tx_m:"max\: %6.2lf %sB" \
  GPRINT:network_tx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_tx_s:"sum\: %6.2lf %sB\n" \
  `# Other area outline` \
  LINE1:network_rx#636363:"in " \
  GPRINT:network_rx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_rx_m:"max\: %6.2lf %sB" \
  GPRINT:network_rx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_rx_s:"sum\: %6.2lf %sB\n"

As always, the manual of rrdgraph explains the possibilities.

I’m using RRDtool 1.6.0 instead of 1.4.8 because I very much prefer its density of x-axis labels. Here are the outputs of our database:

RRDtool 1.4.8: RRDtool 1.6.0:

RRDtool footprint:

Part	Runtime	Memory
Create (9 servers)	0.01 s	1.74 MB
Graph (9 servers)	1.99 s	2.82 MB

Putting it together with Nim

To aggregate the raw data from ServerStatus into 30-second packets I use a small Nim program. It automatically creates new databases when a new server is added and keeps them updated:

import common, json, osproc, os, times, strutils, tables

type
  Data = object
    network_rx, network_tx: BiggestInt
    cpu, memory_used, memory_total, swap_used, swap_total: BiggestInt
    load: float

const freq = 30 # report new data to rrd every 30 seconds

var
  lastUpdated: BiggestInt = 0
  dataTable = initTable[string, Data]()
  count = 0
  countTable = initTable[string, int]()

proc rrdCreate(file, dataSources: string) =
  discard execCmd(rrdtool & " create " & file & " --step " & $freq & " " & dataSources &
    " RRA:AVERAGE:0.5:6:480 RRA:AVERAGE:0.5:42:480 RRA:AVERAGE:0.5:147:960")

proc rrdUpdate(file: string, values: varargs[string, `$`]) =
  var valuesString = ""
  for value in values:
    if valuesString.len > 0:
      valuesString.add ":"
    valuesString.add value
  discard execCmd(rrdtool & " update " & file & " N:" & valuesString)

proc updateServer(server: JsonNode) =
  let domain = server["type"].str

  for name, value in dataTable.mgetOrPut(domain, Data()).fieldPairs:
    when value is BiggestInt:
      value += server[name].num
    elif value is float:
      value += server[name].fnum
    else:
      error "Unhandled type in Data object"

  inc countTable.mgetOrPut(domain, 0)

  # Only save data if we got 30 values in the expected time span
  if count == freq and countTable[domain] == freq:
    let data = dataTable[domain]
    if data == Data():
      dataTable.del(domain)
      return
    else:
      dataTable[domain] = Data()

    let
      fileNet = (rrdDir / domain) & "-net.rrd"
      fileCpu = (rrdDir / domain) & "-cpu.rrd"
      fileMem = (rrdDir / domain) & "-mem.rrd"

    if not existsFile fileNet:
      fileNet.rrdCreate("DS:network_rx:GAUGE:60:0:U DS:network_tx:GAUGE:60:0:U")
    if not existsFile fileCpu:
      fileCpu.rrdCreate("DS:cpu:GAUGE:60:0:100 DS:load:GAUGE:60:0:U")
    if not existsFile fileMem:
      filemem.rrdCreate("DS:memory_used:GAUGE:60:0:U DS:memory_total:GAUGE:60:0:U DS:swap_used:GAUGE:60:0:U DS:swap_total:GAUGE:60:0:U")

    fileNet.rrdUpdate(data.network_rx div freq, data.network_tx div freq)
    fileCpu.rrdUpdate(min(data.cpu div freq, 100), data.load / freq)
    fileMem.rrdUpdate(data.memory_used div freq, data.memory_total div freq, data.swap_used div freq, data.swap_total div freq)

proc updateAllServers =
  let statsJson = parseFile statsJsonFile
  let newUpdated = parseBiggestInt statsJson["updated"].str

  if newUpdated <= lastUpdated:
    return

  inc count

  for server in statsJson["servers"]:
    try:
      updateServer(server)
    except:
      discard

  if count == freq:
    count = 0
    for val in countTable.mvalues:
      val = 0

while true:
  let startTime = epochTime()

  updateAllServers()

  sleep(int(epochTime() - startTime + 1) * 1000) # every second

The final graphs can be seen on the DDNet Server Statistics page.

Nim monitor footprint:

Part	CPU	Memory
Monitor (9 servers)	0.03 %	0.89 MB

Alerts through Cron and Mail

Now we certainly have nice graphs, but automated alerts about suspicious events would be even better, for example:

Network traffic over 2 MB/s for 4 min
Memory and swap over 90% for 4 min
CPU over 90% for 21 min
Load over 10 for 21 min
Server unreachable for 1 hour

To check these conditions a cron job is run regularly and thanks to a MAILTO entry a mail is sent when an alert has been triggered.

We can get out a single value from the database to standard output using PRINT:

rrdtool graph x -s-4min \
  DEF:v=ddnet.tw-net.rrd:network_rx:AVERAGE \
  VDEF:vm=v,AVERAGE \
  PRINT:vm:%lf

The alert program itself is written in Nim as well and merely gets a few values from the databases and checks if the limits are exceeded:

import common, osproc, os, json, strutils

proc get(file, value, time: string): float =
  let (output, errorCode) = execCmdEx(rrdtool & " graph x -s -" & time & " DEF:v=" & file & ":" & value & ":AVERAGE VDEF:vm=v,AVERAGE PRINT:vm:%lf")

  if errorCode != 0:
    raise newException(ValueError, "Error code " & $errorCode & " from rrdtool: " & output)

  result = output.splitLines[^2].parseFloat

if paramCount() != 1:
  echo "alert [1d|7d|49d]"
  quit 1

let statsJson = parseFile statsJsonFile
for server in statsJson["servers"]:
  let
    domain = server["type"].str
    name = server["name"].str
    fileNet = (rrdDir / domain) & "-net.rrd"
    fileCpu = (rrdDir / domain) & "-cpu.rrd"
    fileMem = (rrdDir / domain) & "-mem.rrd"

  template alert(s: string) = echo name, " ", s

  case paramStr(1)
  of "1d":
    if fileNet.get("network_rx", "4min") + fileNet.get("network_tx", "4min") > 2_000_000:
      alert "network traffic over 2 MB/s for 4 min"

    if fileMem.get("memory_used", "4min") + fileMem.get("swap_used", "4min") > 0.9 * (fileMem.get("memory_total", "4min") + fileMem.get("swap_total", "4min")):
      alert "memory and swap over 90% for 4 min"

  of "7d":
    if fileCpu.get("cpu", "21min") > 90.0:
      alert "CPU over 90% for 21 min"
    if fileCpu.get("load", "21min") > 10.0:
      alert "Load over 10 for 21 min"

  of "49d":
    let network_rx = fileNet.get("network_rx", "4410")
    if network_rx != network_rx: # NaN
      alert "unreachable for 1 hour"

  else:
    echo "unknown parameter ", paramStr(1)
    quit 1

Nim alert footprint:

Part	Runtime	Memory
Alert (9 servers)	0.25 s	2.12 MB

Conclusion

Live view of the last day of the DDNet.tw web server, also hosting this blog:

You can see the full graphs on the DDNet Server Statistics page. As usual you can find the entire source code in our git repository.

All in all the system runs on very little resources, puts out some nice graphs and alerts me automatically about defined problems.

Part	Runtime	CPU	Memory
Client		0.14 %	3.70 MB
Server (9 clients)		0.58 %	0.18 MB
Create (9 servers)	0.01 s		1.74 MB
Graph (9 servers)	1.99 s		2.82 MB
Monitor (9 servers)		0.03 %	0.89 MB
Alert (9 servers)	0.25 s		2.12 MB

DDNet Live: Twitch spectates an online game

2016-01-10T00:00:00-05:00

Last night I had an idea and implemented it, soo let’s see what will happen. But first, the idea:

Have a livestream [1] of DDNet running non-stop [2] that always shows some interesting [3] players on the server.

The resulting livestream is running on Twitch. All the scripts are on Github

1. Livestream

It’s surprisingly simple to livestream from Linux to Twitch. Only FFmpeg is needed:

#!/bin/sh

INRES="1280x720" # input resolution
FPS="30" # target FPS
GOP="60" # i-frame interval, should be double of FPS, 
GOPMIN="30" # min i-frame interval, should be equal to fps, 
THREADS="2"
CBR="2000k" # constant bitrate (should be between 1000k - 3000k)
QUALITY="ultrafast"  # one of the many FFMPEG preset
AUDIO_SRATE="44100"
AUDIO_CHANNELS="2" # 1 for mono output, 2 for stereo
AUDIO_ERATE="96k" # audio encoding rate
SERVER="live-fra" #  http://bashtech.net/twitch/ingest.php for list
source ./secret.sh

ffmpeg -v 0 -f x11grab -s "$INRES" -r "$FPS" -i :0.0 \
  -f alsa -i pulse -f flv -ac $AUDIO_CHANNELS \
  -b:a $AUDIO_ERATE -ar $AUDIO_SRATE \
  -vcodec libx264 -g $GOP -keyint_min $GOPMIN -b:v $CBR \
  -minrate $CBR -maxrate $CBR -vf "format=yuv420p"\
  -preset $QUALITY -acodec libmp3lame -threads $THREADS \
  -strict normal -bufsize $CBR \
  "rtmp://$SERVER.twitch.tv/app/$STREAM_KEY"

This works pretty well, but takes quite a bit of CPU on my old computer. So instead I wanted to run it on my new server with a Haswell-era J1900 CPU.

Possible ways to improve performance:

Use an OpenGL recorder instead of x11grab
Get Intel Quick Sync Video working for hardware H264 encoding

2. Running Non-Stop

My server is a small and cheap ASRock Q1900-ITX:

The nice part is that it’s passively cooled and quite power efficient, drawing only 5 Watts in idle. This machine has been running as my home server for quite some time, but barely gets any action. Let’s change that!

The X server starts without problems even without any monitors attached, the only thing that’s left to do is increasing the framebuffer size so that our game can run in it:

$ cat ddnet.sh
#!/bin/sh

pulseaudio --start
xrandr --fb 1280x720 # Adjust framebuffer
cp settings_ddnet.cfg ~/.teeworlds/ # Restore backup
cd ddnet && ./DDNet

$ xinit ddnet.sh

We don’t even need a window manager. After all we just run a single window in exactly the resolution of the framebuffer.

The FFmpeg recording still works in exactly the same way. Unfortunately I couldn’t get H264 hardware encoding to work with the J1900 CPU. Related bug reports make me believe it just doesn’t work on these cheaper Intel CPUs.

3. Artificial Intelligence Twitch Control

Now that we have the game running and are streaming it to Twitch, we need to control it somehow. My goal was to find an approach that always shows some interesting players in action, so that you could watch the stream all day and enjoy it.

But finding a reasonable way to do this seems too complicated and I didn’t look forward to hacking the DDNet C/C++ source code, so instead I opted to utilize an existing system in DDNet: FIFO command input!

DDNet servers and clients can be remote controlled through a FIFO file. This is very useful to send the same commands to dozens of servers at once. But for the client its use was pretty limited, until now!

Instead of thinking of an algorithm to find interesting players, why not let the Twitch viewers themselves control who they want to watch through the chat?

I did not modify the DDNet source code in any way and instead wrote a small Nim script based on the IRC module to connect to Twitch’s IRC server and forward the commands to the FIFO:

import irc, strutils, secret

const forbiddenCommands = @["exec", "quit", "exit", "disconnect"]

var
  client = newIrc("irc.twitch.tv", nick = "ddnetlive",
    serverPass = serverPass, joinChans = @["#ddnetlive"])
  fifo = open("input.fifo", fmWrite)

proc handle(nick, cmd: string) =
  for f in forbiddenCommands:
    if cmd.contains(f):
      return

  echo nick, ": ", cmd
  stdout.flushFile()

  fifo.write(cmd)
  fifo.write("\n")
  fifo.flushFile()

client.connect()
var event: TIrcEvent

while true:
  if client.poll(event):
    case event.typ
    of EvConnected:
      discard
    of EvDisconnected, EvTimeout:
      client.reconnect()
    of EvMsg:
      case event.cmd
      of MPrivMsg:
        handle(event.nick, event.params[^1])
      else:
        discard

Right now there is really not much limitation to what you can do. Only a handful of commands are explicitly blocked, otherwise the client can be controlled freely and every single chat message is sent to the client. The client commands and client settings list the available commands and settings. Chat commands can be sent to the server as well through the say command. Some examples:

connect ger.ddnet.org:8303
team -1 # Join spectators
spectate_next # Spectate the next player
spectate 0 # Spectate player with ID 0
say Hi from twitch.tv/ddnetlive # Write chat messages
player_name DDNetLive # Change name

Let’s see how this goes. Luckily the Nim script runs independently from the server, so I will be able to make changes to it on the fly. Just be nice and don’t cause any trouble, thanks.

Feel free to head over to Twitch, watch the action and control the server using the Twitch chat. Twitch has some delay, so it takes a few seconds for you to see your command executed.

What is HookRace?

2014-07-07T00:00:00-04:00

I have been running DDraceNetwork for 1 year, which started out as a little server of a Teeworlds modification called DDRace. It has turned into the biggest project within Teeworlds, offering servers in 6 countries with thousands of players and dozens of mappers. We also offer a custom server and client (Windows, Linux, OS X, Android).

Now it’s time to start something new. I’m working on a new game called HookRace, which will be the successor to DDraceNetwork. It will be its own game, not based on Teeworlds.

Some of the plans I have so far for HookRace:

Vector Graphics (SVG) ingame for a sharp image on any screen
Whole physics is executed at the client as well as the server, for perfect prediction
No lags with high player numbers
Accounts to prevent faking

As the programming language for HookRace, I chose Nim:

# Game of Life in terminal
import os, strutils, math

randomize()

var w, h: int
if paramCount() >= 2:
  w = parseInt paramStr 1
  h = parseInt paramStr 2
if w <= 0: w = 30
if h <= 0: h = 30

# Iterate over fields in the universe
iterator fields(a = (0,0), b = (h-1,w-1)) =
  for y in a[0]..b[0]:
    for x in a[1]..b[1]:
      yield (y,x)

# Create a sequence with an initializer
proc newSeqWith[T](len: int, init: T): seq[T] =
  result = newSeq[T] len
  for i in 0 .. <len:
    result[i] = init

# Initialize
var univ, univNew = newSeqWith(h, newSeq[bool] w)

for y,x in fields():
  if random(10) < 1: univ[y][x] = true

while true:
  # Show
  stdout.write "\e[H"
  for y,x in fields():
    stdout.write if univ[y][x]: "\e[07m  \e[m" else: "  "
    if x == 0: stdout.write "\e[E"
  stdout.flushFile

  # Evolve
  for y,x in fields():
    var n = 0
    for y1,x1 in fields((y-1,x-1), (y+1,x+1)):
      if univ[(y1+h) mod h][(x1 + w) mod w]:
        inc n

    if univ[y][x]: dec n
    univNew[y][x] = n == 3 or (n == 2 and univ[y][x])
  swap univ, univNew

  sleep 200

Nim is a rather new language, and it took me some time to choose it. What I love about the language:

Python-like syntax
Statically typed, compiled
High performance (same ballpark as C/C++)
Garbage Collector that can be controlled for soft real-time
Produces executables without dependencies (compiles to C)
Easily interfaces C libraries (compiles to C)
Clean and powerful metaprogramming

On this blog I will report about my progress in the development of HookRace and my current Nim adventures.