Frugality Posts · HookRace Blog

€9 Ticket

2022-08-20T00:00:00-04:00

As a relief for rising energy costs and inflation Germany is offering nearly-free public transportation for the months of June, July and August. For just €9 per month you can take any kind of regional public transportation in all of Germany. That’s basically everything except the IC/ICE for long distance. This means that a monthly ticket is now as cheap as what I would normally pay for a single route. In this post I want to talk about my experiences with this €9 ticket.

Update 2022-09-01: The €9 ticket ended yesterday without a similar replacement system in place. So many people are again back to paying about €100 per month for much smaller regional tickets instead (see for example this German Reddit thread. Financially it makes more sense for me personally to go back to driving. (I hope the gasoline in the car is still good after three months.) My hope is that these three months will stay in people’s minds in Germany and shape the future of public transportation to be cheaper and simpler:

Image via Reddit

Train Stations

In the above graphic you can see the train stations in our area. Since we are situated in the Rhine-Neckar metropolitan region the railway network is pretty good here.

(map made with umap, routes made with GraphHopper)

The closest train stop from us is St. Ilgen-Sandhausen at 2 km distance. It offers direct connections to the cities of Heidelberg (160k inhabitants), Mannheim (309k), Karlsruhe (313k), Darmstadt (159k) and Frankfurt (753k). For other directions I prefer going to a different train station since switching times are either too long or there is too much risk of missing the connecting train. Only 89% of regional trains are on time in Germany, with some routes and stations having much worse rates. This is a far cry from Switzerland’s 95% punctuality with an even stricter definition of 3 minutes versus 5 minutes delay in Germany.

Another closeby train station we commonly use is HD-Weststadt/Südstadt at 6 km distance, which is good to cycle and can also be reached via Tram. Occasionally I used Heidelberg Hauptbahnhof as well as Wiesloch-Walldorf, both at slightly under 10 km distance, which are a bit larger and thus are used as stops for Regional-Express trains. These go faster than the Regionalbahn and S-Bahn trains and thus also have fewer stops.

Cycling Trips

The above map contains some of the cycling trips I took by combining them with the €9 ticket. (map made with umap, routes made with GraphHopper)

Last time I posted about cycling and work was about commuting by bicycle for a year. I kept this up and was super happy with this way of getting to SAP’s headquarter in Walldorf and the included exercise. After a while I moved together with my wife and my bike commute lengthened to 10 km each direction, which was still perfectly fine. Then Covid forced me in March of 2020 to work from home I switched to regular cycling trips in the afternoon since I started working early in the morning, in order to have meetings with colleagues in Korea and China. Two 50 km cycling trips per week got me to the same level as the daily commute used to do, and I have a nice route I regularly take, which covers a lot of forest and lake.

In January I switched to a new job at Yugabyte. As the database we develop is distributed, so are the people I work with. Since most meetings fall into US times, I am now working more during the afternoon and evening. This means my cycling trips moved to the morning instead.

Previously I had a job ticket for three years, but other than the €9 ticket it limits the range I can take public transportation in, so that many of my destinations would be barely outside of the covered range. When looking at a map of Germany with all the different transport associations, each with their own tickets, prices and rules, I feel reminded of historical maps of the Holy Roman Empire:

Good luck figuring out what ticket to get to combine with your job ticket. Luckily the €9 ticket changed this, at least temporarily, making life simpler.

I used the ticket during the working days to discover new cycling routes. By cycling one direction and taking a direct train connection back, I can cover nearly twice the distance as usual with the same starting and stopping point. With this method I took many routes I had never taken before, discovering beautiful cycling paths along the Neckar river for example. Luckily the trains are not too crowded during working days, especially outside of rush hour. The cycling usually takes around three times as long as the train ride back.

The image above shows an almost empty train during a working day.

Even with a single train connection things can go wrong though. One time the train in front of ours started burning and so we got delayed for an unknown time. But since I had my bicycle with me, that meant I could just cycle back the rest of the way and get some extra exercise for free. If I was really out of energy I could have waited in the train, but often busses are used as a replacement, and bicycles on busses don’t fit well.

The weather was unusually dry the last months, the last few days had the first few drops of rain I remember since the beginning of June. This is of course bad for nature here, as I saw at the Rhine river, which is much lower than usual. For cycling it’s quite good though, only got wet once.

Jogging

This amount of cycling of course meant that my bicycle had to get some repairs done. While I had it at the repair shop, I tried out jogging from Heidelberg back to my home town Leimen. The route goes through the “Kleiner Odenwald” forest for nearly the entire way, and the rest are some fields. It goes through Heidelberg’s highest mountain, the Königstuhl, with a nice view of the city and surrounding area.

The “Ladder to Heaven” in the beginning with its 1300 steps accounts for most of the elevation and is still too tough for me to do at faster than walking speed. After I got my bicycle back I still kept doing this jogging route a few times per week.

Nearby Excursions

There are lots of cities and nice nature nearby. So we used public transportation for excursions on every weekend. It was noticable that the trains were crowded, and at times people couldn’t even enter the train anymore. When going to the cinema we noticed that we should really catch the last train at 22:30, since the next connection would take all night.

One time the train back from Frankfurt was late, then got cancelled. The next train was also delayed, and now overcrowded because it had twice the passengers. Unfortunately it took me too long to realize that with more than 20 minutes delay, we should be able to take high speed ICE trains as a replacement and get the cost reimbursed by Deutsche Bahn.

You still need to wear masks as a Covid prevention on public transportation in Germany. This is basically the only remaining occasion, other than at the Doctor’s office, where you have to wear masks. Even with all the travel using public transportation and basically no distancing we haven’t caught Covid as far as we are aware based on symptoms and some rapid tests.

Eating Out

My wife picked out many great restaurants in nearby cities for us to try out. By car it would not make sense to just drive to another city for a dinner, but using public transportation we had no problem spending an hour on a train, where you can comfortable talk or surf the web, to reach our destinations. This probably has a really good economic effect on restaurants and cafes, many were so fully packed that we got the last seats or had to reserve beforehand.

Family Visits

For trips to cities and cycling trips we mostly take direct train connections. That usually works quite well. To visit family however the destinations are smaller villages, so the connections are worse and there is no direct connection at all. So a 40 minute drive turns into a 2 hour train ride with 3 different trains instead. Coincidentally this is the same time I need for the route by bicycle.

There is a relatively high chance of missing a connecting train, or of getting stuck in the middle of the route. One time a truck crashed into the train bridge, so our train couldn’t continue. Instead of waiting around for an hour or two we took the train back home and, for the only time during these three months so far, had to switch to the car to keep our timeline. At least the highway was quite empty, there seems to be some conflicting data on whether the €9 ticket actually has an effect on reducing car traffic though.

Getting Visits

While my wife and I didn’t do any long distance travel using the €9 ticket, others did. I had a co-developer of DDraceNetwork visit all the way from Berlin using the €9 ticket. For people who have time it has been much easier to visit others with the €9 ticket, without having to worry about cost.

While on Vacation

While we didn’t use the €9 ticket to go to a vacation, it was still useful during a car-based vacation. While driving to Berchtesgadener Land we saw a train line that seemed to lead to Berchtesgaden. So we stopped at the closeby train station and spontaneously took the train instead, which lead us through beautiful sceneries.

On the way back from the same vacation we had a stop in Munich. Instead of driving the car all the way into the city center I checked for the closest free Park and Ride garage from the highway. After parking the car there, the train was unfortunately cancelled, so we had to take a bus and metro, but still reached the city center quickly.

After having used public transportation a lot back at home, we also felt more comfortable using it on vacation in other countries, even if it costs a bit more there. So on Mallorca we explored the entire island using the extensive bus network instead of renting a car. The photo above shows the most distant bus stop we reached, at a lighthouse at the end of a long curvy road. Next week we are taking the TGV to Paris, which tops out at 320 km/h and manages the 550 km route in just 2:36, an average of 212 km/h. The image below is luckily not the TGV:

Going to the airport by train worked fine, but we had to get there a few hours earlier in case was some train delay causing us to miss the connecting train. On the way back there indeed was a huge delay, with a train line being closed down for hours. After a slow trickle of information from the train operator, we had to figure out an alternative route on our own, since the DB app was still showing routes that could not be operated for the next few hours because of the train line closure.

Conclusion

There are still two weeks left for the €9 ticket and I’m planning to make the best use of them. So far I’ve been taking a trip on most days, so it’s definitely been worth it and will be missed.

Unfortunately the €9 ticket won’t be prolonged. The high public interest has lead to some plans for a relatively cheap follow-up ticket, but again with some regional limitations for the cheap ticket. Other countries are even implementing entirely free public transport meanwhile, so there is some hope for the future.

Linux Desktop Setup

2019-01-15T00:00:00-05:00

Russian Translation by Akhmad Karimov

My software setup has been surprisingly constant over the last decade, after a few years of experimentation since I initially switched to Linux in 2006. It might be interesting to look back in another 10 years and see what changed. A quick overview of what’s running as I’m writing this post:

Motivation

My software priorities are, in no specific order:

Programs should run on my local system so that I’m in control of them, this excludes cloud solutions.
Programs should run in the terminal, so that they can be used consistently from anywhere, including weak computers or a phone.
Keyboard focused is nearly automatic by using terminal software. I prefer to use the mouse where it makes sense only, reaching for the mouse all the time during typing feels like a waste of time. Occasionally it took me an hour to notice that the mouse wasn’t even plugged in.
Ideally use fast and efficient software, I don’t like hearing the fan and feeling the room heat up. I can also keep running older hardware for much longer, my 10 year old Thinkpad x200s is still fine for all the software I use.
Be composable. I don’t want to do every step manually, instead automate more when it makes sense. This naturally favors the shell.

Operating Systems

I had a hard start with Linux 12 years ago by removing Windows, armed with just the Gentoo Linux installation CD and a printed manual to get a functioning Linux system. It took me a few days of compiling and tinkering, but in the end I felt like I had learnt a lot.

I haven’t looked back to Windows since then, but I switched to Arch Linux on my laptop after having the fan fail from the constant compilation stress. Later I also switched all my other computers and private servers to Arch Linux. As a rolling release distribution you get package upgrades all the time, but the most important breakages are nicely reported in the Arch Linux News.

One annoyance though is that Arch Linux removes the old kernel modules once you upgrade it. I usually notice that once I try plugging in a USB flash drive and the kernel fails to load the relevant module. Instead you’re supposed to reboot after each kernel upgrade. There are a few hacks around to get around the problem, but I haven’t been bothered enough to actually use them.

Similar problems happen with other programs, commonly Firefox, cron or Samba requiring a restart after an upgrade, but annoyingly not warning you that that’s the case. SUSE, which I use at work, nicely warns about such cases.

For the DDNet production servers I prefer Debian over Arch Linux, so that I have a lower chance of breakage on each upgrade. For my firewall and router I used OpenBSD for its clean system, documentation and great pf firewall, but right now I don’t have a need for a separate router anymore.

Window Manager

Since I started out with Gentoo I quickly noticed the huge compile time of KDE, which made it a no-go for me. I looked around for more minimal solutions, and used Openbox and Fluxbox initially. At some point I jumped on the tiling window manager train in order to be more keyboard-focused and picked up dwm and awesome close to their initial releases.

In the end I settled on xmonad thanks to its flexibility, extendability and being written and configured in pure Haskell, a great functional programming language. One example of this is that at home I run a single 40” 4K screen, but often split it up into four virtual screens, each displaying a workspace on which my windows are automatically arranged. Of course xmonad has a module for that.

dzen and conky function as a simple enough status bar for me. My entire conky config looks like this:

out_to_console yes
update_interval 1
total_run_times 0

TEXT
${downspeed eth0} ${upspeed eth0} | $cpu% ${loadavg 1} ${loadavg 2} ${loadavg 3} $mem/$memmax | ${time %F %T}

And gets piped straight into dzen2 with conky | dzen2 -fn '-xos4-terminus-medium-r-normal-*-12-*-*-*-*-*-*-*' -bg '#000000' -fg '#ffffff' -p -e '' -x 1000 -w 920 -xs 1 -ta r.

One important feature for me is to make the terminal emit a beep sound once a job is done. This is done simply by adding a \a character to the PR_TITLEBAR variable in zsh, which is shown whenever a job is done. Of course I disable the actual beep sound by blacklisting the pcspkr kernel module with echo "blacklist pcspkr" > /etc/modprobe.d/nobeep.conf. Instead the sound gets turned into an urgency by urxvt’s URxvt.urgentOnBell: true setting. Then xmonad has an urgency hook to capture this and I can automatically focus the currently urgent window with a key combination. In dzen I get the urgent windowspaces displayed with a nice and bright #ff0000.

The final result in all its glory on my Laptop:

I hear that i3 has become quite popular in the last years, but it requires more manual window alignment instead of specifying automated methods to do it.

I realize that there are also terminal multiplexers like tmux, but I still require a few graphical applications, so in the end I never used them productively.

Terminal Persistency

In order to keep terminals alive I use dtach, which is just the detach feature of screen. In order to make every terminal on my computer detachable I wrote a small wrapper script. This means that even if I had to restart my X server I could keep all my terminals running just fine, both local and remote.

Shell & Programming

Instead of bash I use zsh as my shell for its huge number of features.

As a terminal emulator I found urxvt to be simple enough, support Unicode and 256 colors and has great performance. Another great feature is being able to run the urxvt client and daemon separately, so that even a large number of terminals barely takes up any memory (except for the scrollback buffer).

There is only one font that looks absolutely clean and perfect to me: Terminus. Since it’s a bitmap font everything is pixel perfect and renders extremely fast and at low CPU usage. In order to switch fonts on-demand in each terminal with CTRL-WIN-[1-7] my ~/.Xdefaults contains:

URxvt.font: -xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*
dzen2.font: -xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*

URxvt.keysym.C-M-1: command:\033]50;-xos4-terminus-medium-r-normal-*-12-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-2: command:\033]50;-xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-3: command:\033]50;-xos4-terminus-medium-r-normal-*-18-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-4: command:\033]50;-xos4-terminus-medium-r-normal-*-22-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-5: command:\033]50;-xos4-terminus-medium-r-normal-*-24-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-6: command:\033]50;-xos4-terminus-medium-r-normal-*-28-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-7: command:\033]50;-xos4-terminus-medium-r-normal-*-32-*-*-*-*-*-*-*\007

URxvt.keysym.C-M-n: command:\033]10;#ffffff\007\033]11;#000000\007\033]12;#ffffff\007\033]706;#00ffff\007\033]707;#ffff00\007
URxvt.keysym.C-M-b: command:\033]10;#000000\007\033]11;#ffffff\007\033]12;#000000\007\033]706;#0000ff\007\033]707;#ff0000\007

For programming and writing I use Vim with syntax highlighting and ctags for indexing, as well as a few terminal windows with grep, sed and the other usual suspects for search and manipulation. This is probably not at the same level of comfort as an IDE, but allows me more automation.

One problem with Vim is that you get so used to its key mappings that you’ll want to use them everywhere.

Python and Nim do well as scripting languages where the shell is not powerful enough.

System Monitoring

htop works great for getting a quick overview of what the software is currently doing. lm_sensors allows monitoring the hardware temperatures, fans and voltages. powertop is a great little tool by Intel to find power savings. ncdu lets you analyze disk usage interactively.

nmap, iptraf-ng, tcpdump and Wireshark are essential tools for analyzing network problems.

There are of course many more great tools.

Mails & Synchronization

On my home server I have a fetchmail daemon running for each email acccount that I have. Fetchmail just retrieves the incoming emails and invokes procmail:

#!/bin/sh
for i in /home/deen/.fetchmail/*; do
  FETCHMAILHOME=$i /usr/bin/fetchmail -m 'procmail -d %T' -d 60
done

The configuration is as simple as it could be and waits for the server to inform us of fresh emails:

poll imap.1und1.de protocol imap timeout 120 user "dennis@felsing.org" password "XXX" folders INBOX keep ssl idle

My .procmailrc config contains a few rules to backup all mails and sort them into the correct directories, for example based on the mailing list id or from field in the mail header:

MAILDIR=/home/deen/shared/Maildir
LOGFILE=$HOME/.procmaillog
LOGABSTRACT=no
VERBOSE=off
FORMAIL=/usr/bin/formail
NL="
"

:0wc
* ! ? test -d /media/mailarchive/`date +%Y`
| mkdir -p /media/mailarchive/`date +%Y`

# Make backups of all mail received in format YYYY/YYYY-MM
:0c
/media/mailarchive/`date +%Y`/`date +%Y-%m`

:0
* ^From: .*(.*@.*.kit.edu|.*@.*.uka.de|.*@.*.uni-karlsruhe.de)
$MAILDIR/.uni/

:0
* ^list-Id:.*lists.kit.edu
$MAILDIR/.uni-ml/

[...]

To send emails I use msmtp, which is also great to configure:

account default
host smtp.1und1.de
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on
from dennis@felsing.org
user dennis@felsing.org
password XXX

[...]

But so far the emails are still on the server. My documents are all stored in a directory that I synchronize between all computers using Unison. Think of Unison as a bidirectional interactive rsync. My emails are part of this documents directory and thus they end up on my desktop computers.

This also means that while the emails reach my server immediately, I only fetch them on deman instead of getting instant notifications when an email comes in.

From there I read the mails with mutt, using the sidebar plugin to display my mail directories. The /etc/mailcap file is essential to display non-plaintext mails containing HTML, Word or PDF:

text/html;w3m -I %{charset} -T text/html; copiousoutput
application/msword; antiword %s; copiousoutput
application/pdf; pdftotext -layout /dev/stdin -; copiousoutput

News & Communication

Newsboat is a nice little RSS/Atom feed reader in the terminal. I have it running on the server in a tach session with about 150 feeds. Filtering feeds locally is also possible, for example:

ignore-article "https://forum.ddnet.org/feed.php" "title =~ \"Map Testing •\" or title =~ \"Old maps •\" or title =~ \"Map Bugs •\" or title =~ \"Archive •\" or title =~ \"Waiting for mapper •\" or title =~ \"Other mods •\" or title =~ \"Fixes •\""

I use Irssi the same way for communication via IRC.

Calendar

remind is a calendar that can be used from the command line. Setting new reminders is done by editing the rem files:

# One time events
REM 2019-01-20 +90 Flight to China %b

# Recurring Holidays
REM 1 May +90 Holiday "Tag der Arbeit" %b
REM [trigger(easterdate(year(today()))-2)] +90 Holiday "Karfreitag" %b

# Time Change
REM Nov Sunday 1 --7 +90 Time Change (03:00 -> 02:00) %b
REM Apr Sunday 1 --7 +90 Time Change (02:00 -> 03:00) %b

# Birthdays
FSET birthday(x) "'s " + ord(year(trigdate())-x) + " birthday is %b"
REM 16 Apr +90 MSG Andreas[birthday(1994)]

# Sun
SET $LatDeg 49
SET $LatMin 19
SET $LatSec 49
SET $LongDeg -8
SET $LongMin -40
SET $LongSec -24

MSG Sun from [sunrise(trigdate())] to [sunset(trigdate())]
[...]

Unfortunately there is no Chinese Lunar calendar function in remind yet, so Chinese holidays can’t be calculated easily.

I use two aliases for remind:

rem -m -b1 -q -g

to see a list of the next events in chronological order and

rem -m -b1 -q -cuc12 -w$(($(tput cols)+1)) | sed -e "s/\f//g" | less

to show a calendar fitting just the width of my terminal:

Dictionary

rdictcc is a little known dictionary tool that uses the excellent dictionary files from dict.cc and turns them into a local database:

$ rdictcc rasch
====================[ A => B ]====================
rasch:
    - apace
    - brisk [speedy]
    - cursory
    - in a timely manner
    - quick
    - quickly
    - rapid
    - rapidly
    - sharpish  [Br.]  [coll.]
    - speedily
    - speedy
    - swift
    - swiftly
rasch [gehen]:
    - smartly [quickly]
Rasch {n} [Zittergras-Segge]:
    - Alpine grass [Carex brizoides]
    - quaking grass sedge [Carex brizoides]
Rasch {m} [regional] [Putzrasch]:
    - scouring pad
====================[ B => A ]====================
Rasch model:
    - Rasch-Modell {n}

Writing and Reading

I have a simple todo file containing my tasks, that is basically always sitting open in a Vim session. For work I also use the todo file as a “done” file so that I can later check what tasks I finished on each day.

For writing documents, letters and presentations I use LaTeX for its superior typesetting. A simple letter in German format can be set like this for example:

\documentclass[paper = a4, fromalign = right]{scrlttr2}
\usepackage{german}
\usepackage{eurosym}
\usepackage[utf8]{inputenc}
\setlength{\parskip}{6pt}
\setlength{\parindent}{0pt}

\setkomavar{fromname}{Dennis Felsing}
\setkomavar{fromaddress}{Meine Str. 1\\69181 Leimen}
\setkomavar{subject}{Titel}

\setkomavar*{enclseparator}{Anlagen}

\makeatletter
\@setplength{refvpos}{89mm}
\makeatother

\begin{document}
\begin{letter} {Herr Soundso\\Deine Str. 2\\69121 Heidelberg}
\opening{Sehr geehrter Herr Soundso,}

Sie haben bei mir seit dem Bla Bla Bla.

Ich fordere Sie hiermit zu Bla Bla Bla auf.

\closing{Mit freundlichen Grüßen}

\end{letter}
\end{document}

Further example documents and presentations can be found over at my private site.

To read PDFs Zathura is fast, has Vim-like controls and even supports two different PDF backends: Poppler and MuPDF. Evince on the other hand is more full-featured for the cases where I encounter documents that Zathura doesn’t like.

Graphical Editing

GIMP and Inkscape are easy choices for photo editing and interactive vector graphics respectively.

In some cases Imagemagick is good enough though and can be used straight from the command line and thus automated to edit images. Similarly Graphviz and TikZ can be used to draw graphs and other diagrams.

Web Browsing

As a web browser I’ve always used Firefox for its extensibility and low resource usage compared to Chrome.

Unfortunately the Pentadactyl extension development stopped after Firefox switched to Chrome-style extensions entirely, so I don’t have satisfying Vim-like controls in my browser anymore.

Media Players

mpv with hardware decoding allows watching videos at 5% CPU load using the vo=gpu and hwdec=vaapi config settings. audio-channels=2 in mpv seems to give me clearer downmixing to my stereo speakers / headphones than what PulseAudio does by default. A great little feature is exiting with Shift-Q instead of just Q to save the playback location. When watching with someone with another native tongue you can use --secondary-sid= to show two subtitles at once, the primary at the bottom, the secondary at the top of the screen

My wirelss mouse can easily be made into a remote control with mpv with a small ~/.config/mpv/input.conf:

MOUSE_BTN5 run "mixer" "pcm" "-2"
MOUSE_BTN6 run "mixer" "pcm" "+2"
MOUSE_BTN1 cycle sub-visibility
MOUSE_BTN7 add chapter -1
MOUSE_BTN8 add chapter 1

youtube-dl works great for watching videos hosted online, best quality can be achieved with -f bestvideo+bestaudio/best --all-subs --embed-subs.

As a music player MOC hasn’t been actively developed for a while, but it’s still a simple player that plays every format conceivable, including the strangest Chiptune formats. In the AUR there is a patch adding PulseAudio support as well. Even with the CPU clocked down to 800 MHz MOC barely uses 1-2% of a single CPU core.

My music collection sits on my home server so that I can access it from anywhere. It is mounted using SSHFS and automount in the /etc/fstab/:

root@server:/media/media  /mnt/media  fuse.sshfs noauto,x-systemd.automount,idmap=user,IdentityFile=/root/.ssh/id_rsa,allow_other,reconnect 0 0

Cross-Platform Building

Linux is great to build packages for any major operating system except Linux itself! In the beginning I used QEMU to with an old Debian, Windows and Mac OS X VM to build for these platforms.

Nowadays I switched to using chroot for the old Debian distribution (for maximum Linux compatibility), MinGW to cross-compile for Windows and OSXCross to cross-compile for Mac OS X.

The script used to build DDNet as well as the instructions for updating library builds are based on this.

Backups

As usual, we nearly forgot about backups. Even if this is the last chapter, it should not be an afterthought.

I wrote rrb (reverse rsync backup) 10 years ago to wrap rsync so that I only need to give the backup server root SSH rights to the computers that it is backing up. Surprisingly rrb needed 0 changes in the last 10 years, even though I kept using it the entire time.

The backups are stored straight on the filesystem. Incremental backups are implemented using hard links (--link-dest). A simple config defines how long backups are kept, which defaults to:

KEEP_RULES=( \
   7  7 \ # One backup a day for the last 7 days
  31  8 \ # 8 more backups for the last month
 365 11 \ # 11 more backups for the last year
1825  4 \ # 4 more backups for the last 5 years
)

Since some of my computers don’t have a static IP / DNS entry and I still want to back them up using rrb I use a reverse SSH tunnel (as a systemd service) for them:

[Unit]
Description=Reverse SSH Tunnel
After=network.target

[Service]
ExecStart=/usr/bin/ssh -N -R 27276:localhost:22 -o "ExitOnForwardFailure yes" server
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

Now the server can reach the client through ssh -p 27276 localhost while the tunnel is running to perform the backup, or in .ssh/config format:

Host cr-remote
  HostName localhost
  Port 27276

While talking about SSH hacks, sometimes a server is not easily reachable thanks to some bad routing. In that case you can route the SSH connection through another server to get better routing, in this case going through the USA to reach my Chinese server which had not been reliably reachable from Germany for a few weeks:

Host chn.ddnet.org
  ProxyCommand ssh -q usa.ddnet.org nc -q0 chn.ddnet.org 22
  Port 22

Final Remarks

Thanks for reading my random collection of tools. I probably forgot many programs that I use so naturally every day that I don’t even think about them anymore. Let’s see how stable my software setup stays in the next years. If you have any questions, feel free to get in touch with me at dennis@felsing.org.

Comments on Hacker News.

One year of cycling to work

2018-02-20T00:00:00-05:00

Exactly on this day, one year ago, I came back from a one month long trip to Taiwan, went straight to work from the airport and immediately moved into a new apartment after work. Since then I have cycled to work nearly every day.

Fullscreen (map made with umap, routes made with GraphHopper)

Alternatives and motivation

Previously I lived in Heidelberg, 15 km from work and usually took the train, only cycling to work about once a week. But my new apartment was significantly closer to work, only 6 km, which takes just 15-20 minutes to cycle, so it became much easier to use the bicycle to get to work every day.

Taking public transportation instead is possible, and takes about 35-40 minutes. I only do that when I want to go to the city straight after work instead of going home first. For me personally public transportation is more useful in combination with the bicycle on longer trips.

Sometimes I take public transportation to work and just walk back in the evening, which can be great for listening to audiobooks.

For about a month I had borrowed a car and sometimes used it to go to work. Without traffic the driving would take 12 minutes, but with traffic it’s usually 15-25 minutes. Additionally I have to get a parking spot and walk further to the office, so using a car doesn’t save time compared to cycling.

Advantages

I remember feeling tired when getting to work early in the morning using public transportation. After all you just keep sitting and standing, never really jump starting the body. While cycling to work I certainly wake up and feel more energetic and focussed when I arrive.

Having a clear separation of work and private life is important for me, both by using different spaces for each as well as having a strict time separation. So after I get out of work and get onto my bicycle I will still be having work-related thoughts circling in my head. But while cycling I might take a quick stop to write down any insights that I have or tasks that I forgot to finish. Then I can finally shut down and clear my brain from work by enjoying the ride, the views and the physical exercise.

I find the simplicity of cycling for locomotion beautiful. I never have to be stuck in a traffic jam, feeling stressed by moving at a snail’s pace. Even if cycling is slower the feeling of making progress is stronger. I never have to be waiting for a late train, a bus that left too early or be stuck inside of a train while there are problems with the power line. In the worst case I can pick up my bicycle and walk around some obstacle or fix a flat tire in a few minutes. Being outside in nature, at least for half an hour every day, feels good.

Combined with the gym at work I feel like I’m getting enough sports to be reasonably fit without having to spend much time on it. I regularly get home from work at 17:00 with all work and sports done and can enjoy the rest of the day without any worry.

When I’m using the bicycle to get to work already, I feel like the barrier to combining this with larger trips is lowered. I occasionally do 40-50 km cycling tours before or after work, depending on the weather and my mood.

Accommodation

At home my bicycle is locked into a room in the basement, making it reasonably secure and still easily accessible. At work we have roofed-over parking spaces for the bicycles with robust metal bars for locking. Some of the cycle racks even have two floors with an easy mechanism for getting your bicycle down from there. When I don’t need my lock I will just leave it there after work for the next day.

Multiple buildings at work have special showers for cyclists. But to be honest I don’t use them after my usual route to work, for three reasons:

It is cooler in the morning.
The distance is short enough.
I cycle slower when going to work in the morning than when going back.

Especially in the summer it is nice to do a two hour cycling trip at 5 am and go straight to work afterwards. Then the showers are a godsend of course.

The journey is the reward

Close to my home I have to cross a red light, which is also where I had the only accident in this year. A car driver ignored my right of way and so I had to break hard and ended up sitting on their hood. But luckily that didn’t result in any injuries or damages. (Edit: I have never had an accident involving a car before, so this is probably an outlier.)

After exiting the industrial area I am riding exclusively on small field roads. Especially in the morning this makes for nice sunrises with fog on parts of the fields. Along the way there is also a fishing lake.

The way tunnels below train tracks, runs over a stream, tunnels below a large street and finally bridges over another large street. This means there are no large intersections where you have to wait or be cautious. In consequence the ride is quite relaxing.

The only potential stop along the way is a small airfield where planes are occasionally landing and starting. But that’s fun to watch, so I don’t mind waiting for a minute or two.

Numbers

There are about 250 work days in a year in Baden-Württemberg, excluding weekends and public holidays. Of those I was missing for 34 days for vacations, being sick and doing home office. On 11 days I used public transportation, on 5 days a car. That leaves 200 days on which I cycled to work.

The distance to work is 6 km each way, so in total I cycled 2400 km just to get to work and back. I need 20 minutes to work and 15 minutes back, so 117 hours were spent cycling.

I had two flat tires, one accident and needed one major part replacement in the entire year. But to be fair I also cycled in my free time and it was the first large replacement my bicycle needed since I started using it 15.000 km ago.

Animal nature

On one day in summer I counted the wild animals that I saw on my short trip to work, which included 4 rabbits, 1 deer with fawn, 1 white stork with young in the nest as well as a group of 21 white storks in the fields. A few weeks ago I was happy to see an elusive black stork flying towards the closeby forest for the first time. On another occurence there was a Heron successfully catching a snake.

Apart from wild animals there are of course many people walking their dogs, and the pet home on the way features lots of dogs behind the fence as well as a few cats walking around outside.

Winter is coming

Appropriate clothing is important since cycling exposes you to wind and precipitation.

In summer a t-shirt and short pants are usually fine. For colder times dual-function pants for office and cycling are great, otherwise jeans also work.

During rains I certainly don’t want to wear jeans. Then I also need to bring another set of clothes to switch to once I’m in the office. The most important measure for rainy days is to have good timing, since I have flexible work times and the rains are usually not equally strong for hours. Weather forecasts can be remarkably inaccurate though.

I use wind-proof jackets and shoes as well as gloves and a balaclava in winter. Snow and ice are rather rare and not so strong here, so they are not a big issue. The above photo was taken two years ago and that represents about the worst winter conditions I ever had to cycle in.

Conclusion

I’ll keep cycling to work in the foreseeable future. I don’t think this post is of much use to anyone, but I had fun reflecting my last year of cycling to work. Thanks for reading. Comments on Hacker News.

Broken Hardware, Fixes and Hacks over 8 Years

2016-06-21T00:00:00-04:00

After reading the feedback of my recent article about running DDNet, I noticed that people found it interesting how I’m trying to minimize money and resources. I also noticed that I had been doing something similar with my personal computing hardware setup for an even longer time.

I’ve mostly been using the same hardware for personal computation purposes over the last 8 years. In this article I want to talk about some of the hardware I’ve been using, how it broke and how I fixed the problems or worked around them. My goal was to be frugal about hardware, to keep using the same hardware for a long time and repair it when possible instead of simply buying new hardware.

The reason for this post is that I’m moving and abandoning my old hardware setup. There may be some interesting tales in here. Depending on how you value your time it is probably cheaper to simply buy new stuff instead of repairing old ones, but I consider it a fun activity and a much more rewarding experience.

Whether you call it planned obsolescence or just cheap manufacturing, it’s a fact that electronic hardware tends to break rather quickly. Most reviews don’t take any note of this and so it is difficult to find out which hardware to buy if you value durability. The best I can do is tell you what you should and should not have bought 8 years ago.

For each of these topics there are probably much more detailed reports and guides online, so I will keep it short and just give an overview. If you’re interested in fixing your own hardware, a search engine of your choice is your friend.

Tools used

The tools I use are absolutely simple: Mostly I just used a Weller WECP-20 for soldering, regular screwdrivers and some other, older hardware to steal replacement parts from. Nothing fancy and I don’t claim to do anything special with it.

Liquid-Crystal Displays

Let’s start with what in my experience was the most common problem in consumer electronics, resulting in their failure: broken capacitors

Or at least it was a major problem for me, probably because much of my failing hardware was produced around the time of the capacitor plague.

On the right picture you can see difference between a good capacitor (left) and a bulging capacitor (right).

I used a BENQ FP91GP display for the entire time. At some point I also added a used Samsung SyncMaster P2250.

Both of them failed at some point, stopping to display anything and making high-pitched sounds instead. So of course the first thing I did was open them up. In both of them I found the exact same thing: Bulging capacitors

When you open up hardware you have to be careful about electric shocks. I never received one, but especially power supply units (PSUs), which were the broken parts inside of my displays, can store a high amount of energy for a long time after disconnecting them from power.

I didn’t want to buy new capacitors, even though they are available for low prices and you can even buy a full repair kit online for commonly failing displays. Instead I went through even older hardware in the house and scavenged still functioning capacitors. Mostly you have to look out for three things:

New capacitor has correct characteristics
New capacitor fits into space of the old one
Install the new capacitor the right way or more hardware might blow up (negative stripe on capacitor to white mark on board)

Once I found replacement capacitors I desoldered the broken ones and replaced them. Surprisingly these capacitors from TVs from the 80s still work just fine. My plan was to buy brand-new capacitors once they fail, but it never happened.

In hindsight it’s certainly curious to repair old hardware with parts from even older hardware.

Power Supply Units

The same story didn’t just happen with the PSUs inside of both of my displays, but also with the power supply unit of my computer. I fixed it in the same way and the PSU still works to this day. But since my LC Power LC6420 was never very efficient I bought a replacement anyway, the more efficient HuntKey Jumper 300G.

Unfortunately the new PSU turned out to be too loud for my taste, so I opened it up immediately and noticed that the fan was fixed to 12 V. I reconnected it to 5 V instead and tested that the PSU does not overheat with my system even at high load. I wouldn’t recommend doing this since it immediately voids the warranty and the PSU can overheat easily. Ideally just buy a modern PSU which controls its fan properly based on the internal temperature.

Computer Case

I wasn’t just annoyed with the noise of my PSU though. In the same vein I also switched all 7 case fans of my NZXT Zero from 12 V to 5 V, thus reducing the amount of noise significantly. The fix is simple and does not require any soldering. Instead I used a molex extension cable for the fans and switched its 12 V line to 5 V

The trick here is to remove the pins from the molex cable with a small screwdriver, and switch them around so that 12 V and 5 V are switched, which is quite simple. I just had to make sure that my computer stays cool enough, but with 9 fans in total running inside of it that was no problem, even in the nearly noiseless state of running at 5 V.

The CPU cooler can be controlled by fancontrol on Linux and only spins up when the load is high, otherwise the CPU can be cooled passively, just using the slight flow of air from the case fans.

Random Access Memory

Finally let’s get to an entirely different problem, for which I didn’t even suspect a hardware fault: I’ve been using Gentoo for nearly the entire time, which means I ended up compiling my own programs and libraries. Compiling a single large-scale C++ library failed again and again, at seemingly random positions. The g++ compiler always complained about internal compiler errors. At first I assumed this was a bug in the GNU Compiler Collection, so I switched to other versions, but got the same error in each of them. At some point I noticed that the memory usage increases to multiple GBs when compiling this library and that’s when it hit me: My main memory might be defective and with most programs I just don’t notice.

I booted into Memtest86+ to run the memory benchmark and, lo and behold, a single byte of my RAM was broken. These were my choices on how to fix this:

Remove the RAM module, but then I don’t get the speed benefit of dual channel and am missing out on 2 GB of RAM
Buy a replacement module, but that costs money
Tell the Linux kernel to ignore the broken part of the memory and use the rest of the module

Of course I went with choice 3. A simple Linux kernel parameter memmap=1$0x0007cec2d74 in GRUB marks 1 byte at the address 0x0007cec2d74 as reserved ($), preventing the system from using it as regular main memory. A reboot later everything worked again and I could compile the C++ library. It’s possible that further parts of this memory module will break in the future and you have to repeat this process to disable them as well, but so far this has not happened.

Working around hardware problems with software is cool. And it’s not like I notice a single byte of memory missing.

Graphics Processing Units

My graphics cards like to break. I don’t really play much except for some 2D games, so a good GPU never mattered to me. The GeForce 8500 GT started to show graphics errors after a few years, even while booting, which led me to believe that the problem was on the hardware side and not software related.

Putting the GPU into an oven and baking it for a while fixed the problem for a few days by resoldering cracked solder points, but then it returned. This is probably related to the switch to lead-free solder in 2006, which is more brittle.

I didn’t want to keep baking my GPU, so I used another GPU I had lying around, which someone else had thrown away: A Radeon HD 4350

Mouse

The Logitech MX518 did its job great for the entire time. About 1-2 years ago the left mouse button stopped working reliably though. When keeping it pressed it sometimes released on its own at unpredictable moments. That’s a huge problem while playing something like DDNet.

The first step was of course to open up the mouse. What I saw was that the plastic of the mouse buttons was worn out to the point that it could barely press down the actual button inside of the mouse. I reinforced both mouse buttons with a small piece of thin cardboard. Since then the mouse has not made any further problems and the buttons work absolutely reliably.

Keyboard

I’m using a cheap Logitech Ultra Flat-X keyboard, which doesn’t feel terribly different from a good Thinkpad keyboard. I replaced the green LEDs with blue ones, but that was just an aesthetic change and was easily done with some more soldering.

Unfortunately dust and dirt always accumulate under the keys of a keyboard. If this keeps going for long enough some keys become unusable, so I took out all the key caps every year or so to wash them and clean the space below them. Unfortunately the scissor mechanism on cheap keyboards is quite fragile, so you have to be careful not to break your keys. I don’t need a numpad, so I used the numpad keys as replacements.

After a few years of usage the key labels wear off. So instead I sanded the key labels off entirely, leaving clear key caps which look much better even after many more years of usage. Unfortunately by now the key caps have become so thin that they are starting to break apart, so it’s definitely time for a new keyboard.

Printer

This is the oldest piece of hardware in the list so far. This HP LaserJet 4L has been in continuous use for about 23 years. The only serious thing you need to worry about with it is to have fresh toner cartridges if you plan to print extensively. Luckily I got 10 full toner cartridges on eBay for 1 €, taking care of this problem.

The only part of the printer I had to replace was the pickup roller, which became unable to pull in any pages. I tried cleaning it, but in the end decided to just buy a cheap replacement part, which is easy to find in online shops, even 23 years after the introduction of the printer.

Headphones

My headphones were not really broken, but rather are a chimera. The base are cheap Superlux HD-330 headphones (29 €), a not-so-subtle clone of the Beyerdynamic DT-770 Pro (137 €).

Nevertheless they still sound fine to me, but I don’t claim to be an expert. Since I didn’t like the cable I used the old nylon cable from my previous headphones and soldered them in as replacements. A little bit of soldering gets you quite far.

Instead of using the included fake leather ear pads I bought the original ear pads from Beyerdynamic (21 €), which cost nearly as much as the rest of the headphones.

Amplifier

Finally here’s the oldest part of my setup: A Marantz PM710DC amplifier from 1980 and HECO Superior 700 speakers from around 1985. The speakers still sound just fine. The amplifier on the other hand is showing its age:

The analog potentiometers are filled with dust and you hear a crackling sound when adjusting the volume. Blowing on them with some compressed air and moving the potentiometer a few times entirely removes the crackling for me. In more serious cases potentiometers can also be cleaned with contact cleaner or simply be replaced with some soldering.

Desk Light

The IKEA Global work lamp is quite useful, as it can be attached to many surfaces with a clamp and can be moved flexibly. Unfortunately all the flexible movement kept weakening the plastic connection between lamp and metal arm. In the end the plastic simply snapped off. My plastic glue didn’t work with this kind of plastic, so I decided to work around the problem instead.

Using a few pieces of paper as a buffer the lamp can be inserted into the clamp directly. Now I have the lamp fixed directly above the desk instead of being able to move it around freely, but that’s not a big problem because the entire desk is still well-lit.

Network Cables

I used to have lots of old Ethernet cables, some working only up to 100 Mbit/s, others not working at all at times. Instead of buying many new Ethernet cables, I opted to buy a 100 meter roll of high quality CAT5e Ethernet cable, a hundred RJ45 jacks and a crimper tool. Finally I can build as many cables as I need with the exact lengths I prefer.

Laptops

Initially I had a used Thinkpad T43p from 2005. After a few years of running Gentoo on it and compiling too many packages the fan broke. Fortunately replacement fans are easy to find and opening up and repairing old Thinkpads is a blessing. Printed on the bottom of the laptop you are instructed how to open which part of the laptop. There are official guides for replacing any part of the laptop.

Since 2010 I’m running a used Thinkpad x200s. The only problem it ever had was a broken battery. But since the laptop features replaceable batteries installing a cheap replacement battery from eBay is a matter of seconds.

Home Servers

Broken Thinkpads make good servers. I had an old, very broken Thinkpad T42 lying around. The screen did not work and it could only be powered from the battery, while it didn’t run on AC power at all and couldn’t charge the battery either. My solution was to remove the screen entirely and build a small adapter with some soldering to connect AC power to the battery connection directly, leaving out the battery.

This worked as a home server for a few years, until the performance of such an old Pentium M system got too low and I switched to a totally passive Q1900-ITX system in 2014.

Conclusion

All in all I would say that the hardware held up reasonably well and when it broke it was usually possible to fix it without replacing the entire device. More repairable hardware would still be a huge bonus, especially since the industry has been moving in the opposite direction with ever smaller and more compacted devices.

Maybe someone who read this post found it interesting and will have fun fixing their own hardware (or trying to) once it starts malfunctioning.

I wrote this up on the exact hardware described in this post, while waiting for my new hardware to arrive in the next days. I hope build quality and repairability of PC hardware haven’t gone down in the last years. But only time will tell.

Discussion on Hacker News and r/Hardware.

Experiences of Running an Online Game for 3 Years

2016-06-09T00:00:00-04:00

It’s been roughly 3 years since DDraceNetwork (DDNet) started in the summer of 2013. Last year I wrote a non-technical History of DDNet. Today in this post we will dive into the technical side of what makes DDNet run.

For the uninitiated, DDNet is an open-source modification of Teeworlds, a retro multiplayer shooter. DDNet is a game in which you, instead of killing each other, cooperate with each other to work your way through challenging maps, trying to beat the map or get a better time than other teams.

What we offer for DDNet is the client and server software as well as official servers that run all around the world. We have an international community of players trying to beat the latest maps and records, and dozens of mappers who send in their newest creations for our consideration. A new map is released every few days and occasionally tournaments happen in which the best compete against each other on brand-new maps.

At the time of writing there are 968 people playing Teeworlds, 678 of which are on a server running the DDNet mod, and 430 on the official DDNet servers. So this project is not especially big and not many people stumble upon DDNet. Our player base consists in a large part of old Teeworlds players and their friends who heard about us by word of mouth.

Nevertheless I hope that the technical challenges we faced and face as well as our solutions are interesting. Let’s start with an overview of what we have:

Overview & Financing

DDNet runs official servers in Germany, Russia, USA, Canada, Chile, Brazil, China, South Africa and Iran.

Since DDNet is totally free, has no advertisements and no in-game purchases, it offers no stream of revenue and is solely funded by donations, donated servers and my own money. So one of the goals is to keep costs down: On average we pay 10 € per month for each location.

Let’s see what we can offer with this to our thousands of players (detailed statistics) and how DDNet evolved to keep with increasing number of players, maps and the product of both, ranks.

Servers & Locations

We exclusively use cheap virtual private servers (VPS). They offer enough performance for us and are significantly cheaper than dedicated servers. To have a bit stronger guarantees on performance I prefer KVM and XEN over OpenVZ, but there are good and bad hosters for each. In the end you always have to try out a hoster for a few days up to a month to find out if it is suitable for hosting official DDNet servers. The most important criteria for us are:

Good and consistent CPU performance: should do about 400 players per CPU core
Low network latency to our players in the region: DDNet is only enjoyable with a low and stable ping
0.5 to 1 GB of RAM is enough for ~~everyone~~ most locations: running about 20 game servers per location

When searching for hosters in a country you should prefer to look in the language of the country instead of English. These tend to be cheaper and less overcrowded by international buyers. Be prepared to use a translator and have trouble with the support, but after some time all these hoster websites start to look the same to you, no matter if they’re in English, German, French, Spanish or Farsi.

When in doubt, ask locals what hosters they use or recommend, or check the whois entries for servers in the region that seem to run well. For example the whois 31.186.251.128 entry tells you that DDNet.tw (and this blog) are hosted at Nuclear Fallout Enterprise, connected by the InterNAP Network:

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '31.186.250.0 - 31.186.251.255'

% Abuse contact for '31.186.250.0 - 31.186.251.255' is 'noc@internap.com'

inetnum:        31.186.250.0 - 31.186.251.255
netname:        INAP-LON-nuclearfallout-31-186-250-0
descr:          Nuclear Fallout Enterprise
country:        DE
admin-c:        INAP-RIPE
tech-c:         INAP-RIPE
status:         ASSIGNED PA
mnt-by:         INAP-MAINT-RIPE
created:        2013-08-08T19:09:47Z
last-modified:  2014-03-06T23:07:27Z
source:         RIPE # Filtered

person:         InterNAP Network Operator
address:        InterNAP Network Services
address:        Two Union Square
address:        601 Union Street Suite 1000
address:        Seattle, WA 98101 USA
phone:          +1 206 441 8800
fax-no:         +1 206 256 9580
nic-hdl:        INAP-RIPE
mnt-by:         INAP-MAINT-RIPE
created:        2002-03-04T12:57:06Z
last-modified:  2002-03-04T12:57:06Z
source:         RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.87.3 (DB-2)

Unfortunately the story doesn’t end here for us. Since DDNet is a multiplayer open-source game we seem to attract the kind of people who know how to find a DDoS (distributed denial of service) booter. This resulted in us being kicked out at many hosters since they either do not have any (D)DoS protection or it is not sufficient against some attacks, which then also affect other customers of theirs.

In the end we settled on NFOservers for our main servers, While they do not advertise any DDoS protection, they turned out to be the only one of dozens of hosters I tried that can withstand most of the attacks we get regularly. I’m of course talking about cheap VPSes, I hear that good DDoS protection exists for a few hundred euros per month, which is far outside of our reach. Unfortunately NFOservers is only available in USA and Germany.

Here’s what the attacks in a regular 2 week period looks like in our most popular location, Europe (GER server):

In Russia (Moscow) we had ok experiences with reg.com, at least the ping for local players is lower than in any other location.

Recently we got a server hosted in Iran again, which has always been a difficult country to host a server in, for matters of cost as well as their preference of an intranet over the internet.

In Chile we’re now with zGlobalHost, one of the few hosters with unlimited bandwidth and by far the best I found for just 9 € per month. Their server room looks very cozy:

In Chile we also had this bizarre story back in January, after the entire data center of ZGlobalHost went down:

January 26, 16:04 (local time Chile)
Our provider GTD has trouble again and is still not giving estimated time of repair. We will inform you with more news as soon as possible.
January 26, 18:06
Our provider informs us that in the next 30 minutes everything should be operational again.
January 26, 19:09
GTD reports that there are two fiber cuts, one in Pudahuel and another in the Eastern Zone. They are certifying the fibers and lifting services one by one.
January 26, 22:34
Be informed that when we are back online and GTD reports that it will provide further details about the incident as it was just reported that the fiber cut was due to vandalism, the streets in Providencia are now reported to be safe for repair works again.
We do not have backup lines. It will at least take another hour but repairs should not take longer than until this morning. You will be mailed a report instantly when the servers are back online and tomorrow GTD will send a report with details.
January 26, 22:36
The new details we have say that there were 4 cuts today, which is why there was such a long delay:
Locations: Pudahuel, Providencia, La Florida, Macul
Causes for each cut: Fire, Vandalism, Highway Accident, Material fatigue in electric pole cut it down
All ISPs in various areas have been affected, particularly fiber optic, both business and home addresses.
January 27, 03:00
Service is fully operational again since 02:30 AM.

To make sure that our servers keep running well, we monitor them and record the server statistics. When an unexpected event happens, like a server downtime or high network traffic, a notification is automatically sent. You can read more about this system in a separate post from last month.

Software Platform

As the operating system our servers run Linux, preferably Debian, but also some instances of Ubuntu and CentOS. In areas like South Africa, South America and Iran bandwidth can be exorbitantly expensive, so when you get a donated server in one of those locations you can not afford to be picky.

For the databases we use MariaDB, the drop-in MySQL fork started by the original MySQL developers after Oracle acquired it. We use this database to record our maps, saved games and the ranks of all players worldwide and keep these synchronized around the world.

Initially we started with a single VPS and a database running directly on it. As we grew to add more locations around the world we used a circular master-slave replication. With time some servers turned out to be less stable than others, so they were taken out of the loop, their game servers communicating with nearby MariaDB servers instead:

Now we’re switching away from this circular master-slave replication to having a small number of master databases which are replicated by slaves at each location. This has the advantage that database reads happen locally, so getting your own rank is always fast. Meanwhile new ranks are sent to a master server when possible or otherwise stored in a local file to be added later. This is necessary since our cheap servers are not always connected to the entire internet, sometimes they are only available from inside their own country.

The DDNet client and server, being based on Teeworlds, are also written in C/C++ (also known as C with classes). Luckily their performance was good from the start, but a few optimizations still helped with keeping the load low when running up to 60 game servers on a single VPS, for example:

Reduce the number of syscalls by caching the value of gettimeofday() until a new tick happens or network packet comes in
Do not execute the main game loop when the server is currently empty, as nothing will happen with no players anyway
Use CPU (nice) and IO (ionice) priorities to prevent background tasks (lower priority) to get in the way of game server processes (high priority)
Each VPS compiles its own server binary tuned for the specific target architecture
Prefer to download map files over HTTP from our web server instead of the built-in UDP protocol that can be shaky and slow. This also reduces the network traffic on game servers, keeping network jitter lower.
Instead of downloading the entire archive on a new client release, update only the changed files

Generally the idea is to wait until you notice that some resource is being used too much, benchmark and analyze what the reason for it is, and only then optimize this.

Website

Back when DDNet started in 2013 the website ran on the same VPS as our game servers. Later we switched to hosting the website on a separate VPS, in part to improve the DDoS protection by having mostly UDP on game servers and mostly TCP on the web server, but also to make sure that the website traffic and CPU usage do not impact the game servers.

This has become even more important since the map downloads are now also happening over HTTP from the web server instead of going over UDP from the game servers. Automatically updating the list of available maps is done with a small Nim script that I already presented in a previous article:

#!/usr/bin/env nimscript

import os, crc32, strutils

const
  baseDir = "/home/teeworlds/servers"
  mapdlDir = "/var/www-maps"

for kind, path in walkDir baseDir/"maps":
  if kind != pcFile:
    continue

  let (dir, name, ext) = splitFile(path)

  if ext != ".map":
    continue

  let
    sum = crc32FromFile(path).int64.toHex(8).toLower
    newName = name & "_" & sum & ext
    newPath = mapdlDir / newName
    tmpPath = newPath & ".tmp"

  if existsFile newPath:
    continue

  copyFile path, tmpPath
  moveFile tmpPath, newPath

Most of the DDNet.tw website is statically built by jekyll and automatically deploying using GitHub’s webhooks. That’s a simple solution and requires very few resources.

Pages about player status, ranks, map releases and mappers are also statically built, but by Python scripts directly on the server. This also makes sense since these pages are expensive to build and are requested more often than they have to be generated.

We also used to generate the player pages in the same way, but since we have about 90,000 ranked players so far this would’ve become a bit too expensive. Instead we run a small uWSGI Python server to dynamically generate them now from the data kept in memory. Right now it takes about 40-80 ms to generate a single player page, by far fast enough for us. If the computation time goes up or we suddenly become much more popular caching could be implemented.

We used to package a growing selection of skins with the DDNet client, but maintaining this became too cumbersome and increased memory usage for the client. Now instead the website offers a database of skins where you can select individual skins or skin packs.

Software Releases

DDNet has been in active development for the last 3 years, with our first client & server software release back in October 9, 2013.

Initially I used to build the releases manually using virtual machines for Windows and Linux, later also Mac OS X. After some time this got tedious of course, so I wrote an automated build script. It uses a Debian chroot for Linux building, MinGW for Windows and a QEMU VM for Mac OS X. I set up the Mac OS X QEMU image using a continuously updated guide and it works just fine. I connect to the guest OS through a simple ssh connection that is forwarded to the virtual machine:

# Start the Mac OS X VM
qemu-system-x86_64 [...] \
  -netdev user,id=hub0port0,hostfwd=tcp::10022-:22 \
  -device e1000-82545em,netdev=hub0port0,id=mac_vnet0 \
  &>/dev/null &

# Wait until VM is booted up
while ! ssh -p 10022 -o ConnectTimeout=10 localhost exit; do true; done

# Run build script
ssh -p 10022 localhost "# Run build script for Mac OS X"

Building for Android is also possible but disabled right now because we still have to port the Android version to SDL2, and DDNet is not really playable on Android anyway since a mouse is the most important tool in the game.

When you think about it, it’s pretty amazing that you can easily cross-compile on a single machine for all of these targets:

Windows x86, x86_64
Linux x86, x86_64
Mac OS X
Android armeabi, armeabi-v7a, mips, x86

The automated builds happen on my cheap ASRock Q1900-ITX home sever (which was also used to stream gameplay from DDNet servers) and still only take a few minutes to complete:

Finally when all goes well you get a small summary of the build times that looks like this:

Preparation:      10 s
Linux x86_64:     62 s
Linux x86:        64 s
Windows x86_64:  106 s
Windows x86:      84 s

For the automated builds we use bundled static libraries, while people who build our software locally probably prefer to use their system libraries. Using pkg-config we detect whether libraries are available at compile-time and otherwise use the static ones if they are available. This behaviour can be controlled manually as well:

# Analog to ./configure
bam config curl.use_pkgconfig=false \
           opus.use_pkgconfig=false \
           opusfile.use_pkgconfig=false \
           ogg.use_pkgconfig=false
# Analog to make
bam release

Even a JavaScript port of DDNet client is available for playing without any installation, compiled with Emscripten. For this purpose the official servers also accept WebSocket connections additionally to UDP.

New releases are added to the website with a nice changelog. But most existing players get their update notification in the client directly and run the updater from there, only downloading the files that actually changed instead of the entire new release.

Map Testing & Releases

The best part about DDNet is its active community. New maps for our servers are regularly sent in to be tested and later released. Initially the testing process was implemented as Trac installation. But it turned out that the non-technical people did not appreciate it much, so we switched to a regular phpBB forum for testing and communication.

New maps can be uploaded by our testers to the test servers and then tested there. The map upload happens with an upload script to the web server. The test servers around the world use sshfs to access these maps.

Once a new map has been cleared by the testers and all bugs fixed by the mapper, it can be released on the proper server. The preparation for this happens on the web server and is then synchronized to the other servers using an internal git repository.

The new map is also announced on the recent map releases page as well as its feed. Players on the official servers are informed about the map release using a broadcast message.

It’s possible to take a look at our maps using a WebGL renderer that supports parts of the map format, but no animations for example: Lonely map

When a popular new map is released, it’s possible to have hundreds of players taking up the challenge of playing the new map at once, either in small teams or in big groups.

Finally every day the newly released maps are added to our public maps repository, which can be used to easily run a clone of the official DDNet servers at home or on your own server. Just download the repository, add the DDNet-Server binary to the same directory and run it. A simple shell script suffices to update this repository:

#!/usr/bin/env zsh
mkdir -p /home/teeworlds/ddnet-maps
cd /home/teeworlds/ddnet-maps
rm -rf types

echo 'add_path $USERDIR\nadd_path $CURRENTDIR' > storage.cfg

for i in `cat ../servers/all-types`; do 
  TYPEDIR=types/${i:l}
  echo "add_path $TYPEDIR" >> storage.cfg
  mkdir -p $TYPEDIR/maps
  grep "|" ../servers/$TYPEDIR/maps | cut -d"|" -f2 | while read j; do 
    cp -- "../servers/maps/$j.map" $TYPEDIR/maps
    cp ../servers/$TYPEDIR/flexvotes.cfg $TYPEDIR
    grep -v "flexname.cfg" ../servers/$TYPEDIR/flexreset.cfg > $TYPEDIR/flexreset.cfg
    tail -n +5 ../servers/$TYPEDIR/votes.cfg > $TYPEDIR/votes.cfg
  done
done

git add * &>/dev/null
git commit -a -m "daily update" &>/dev/null
git push &>/dev/null

Server Types

In the start DDNet hosted only new maps categorized by their difficulty:

Later we also added categories for maps from old servers:

Oldschool: Ancient maps
DDmaX: Old DDRace server before DDNet

As well as categories for different kinds of single player maps:

Solo
Race: Classic mod with fewer features, converted maps to new DDNet format
Dummy: Maps where you play with another player that is unable to move

The idea is that, since we are the major hoster in many countries, many of these good maps would be forgotten forever if we don’t host them. At the same time we try to motivate people to make creative new maps, having worked together with mappers to add new features that they require.

While the DDNet servers are the main focus, we also use our servers for hosting a few other Teeworlds mods since they use very few resources and the servers are running already anyway.

Tournaments

When a well-known mapper makes a special map, they can choose to have their map played at an official DDNet tournament. For this it is necessary to keep the map under wraps, showing it only to a few select testers. This ensures that none of the players at the tournament will be familiar with the map already, which would give them an unfair advantage.

Usually tournaments are done on Sunday, 20:00 Central European Time since most of our player base is located in Europe. You can read about the work that goes into preparing a tournament. But the really stressful part is actually holding the tournament.

Since it runs at the same time on all (currently 9-10) servers worldwide, you need to control them all at once. If the tournament map has not been tested well enough it might need to be quickly fixed and reloaded during the tournament. Sometimes problems occur on a single server. Unfortunately tournaments also attract DDoS attacks.

Here you can see the last tournament’s top run, for which the players had 2 hours to compete:

Some of our mappers even created fancy ingame loading screens for the start of the tournament:

Client & Server Development

Development of the client and server software is often challenging because of backwards compatibility:

People with regular Teeworlds client should still be able to play on DDNet
People with old DDNet client (or custom client) should still be able to play
Old ranks would be invalidated by slight fixes/changes in game physics

Nevertheless we managed to put in many new features, for example:

64 player servers instead of just 16 players:
Players with vanilla Teeworlds client only see the 15 closest players on a server.
Protect players from connection timeouts:
The timeout protection works by the client having a unique timeout code that it sends to the server whenever it connects. This tells the server if the player has had a timeout before and they can get back their ingame character.
Allow saving game in team and continuing at a later time:
The save games are stored in our MariaDB database, but can only be loaded at a single location. Otherwise players would have the ability to load a save game multiple times, which is not what we want.
Save the ranks of teams and not just single players
SDL2 client instead of old SDL1.2
New physics: Jetpack, extended teleporters, walljump, tune zones
Highly compressed Opus sounds in maps, it’s amazing how much quality you get with small bit rates
In-client updater using HTTPS and a simple JSON file for control
Dummy feature to control two ingame characters at once:
At first this was meant to help map testers to be able to test maps easier, without requiring a second player. But it’s also very popular with regular players as well as mappers who come up with interesting maps where only one player has all the control.
…

Particularly important for the automation of the official DDNet servers is our FIFO console system. Every server listens for commands from a FIFO file. This enables us to automate the map releases, server restarts and broadcasts among other things. To broadcast a message about a new map release to all servers, you can just do this:

echo 'broadcast "New map Uniswim by Im corneum just released on Solo server"' > servers/*.fifo

We have a small Python script that can do this automatically and also broadcast in a custom big ASCII art font:

Using the FIFO system at multiple locations at once is achieved using Cluster SSH.

We also offer a list of official DDNet servers in JSON which is used to inform the client of the official servers as well as internally to generate the status page.

Unfortunately attacks with spoofed IP addresses are quite common for us. So we have added a spoofing protection using tokens to prevent attackers from filling up servers with players with spoofed IPs.

We have no automated tests for the client and server, the community quickly catches new bugs though. At least we use automated Circle CI building to make sure new changes still compile fine.

Communication

Ingame communication is a bit tricky since the game is built on simplicity and we prefer not to have accounts. Introducing accounts this late into existance would give another vector of abuse by giving attackers the chance to register other people’s names. So you can never be sure who you’re actually talking to. Thanks to some Social Engineering this has been a common source of leaked moderator passwords.

So for official communications the DDNet forum is used instead. We tried to use Let’s Encrypt for SSL/TLS certificates for the website, but it turned out that Windows XP is not supported and many of our players have old systems still running Windows XP. I guess that’s in the nature of running one of the few actively developed online games that work perfectly fine on 15 year old machines. Instead we’re now using StartSSL, which seems to work fine, except that wildcards would cost money. Once Let’s Encrypt works fine on Windows XP we might be able to switch back.

Otherwise technically inclined people mostly discuss on IRC (#ddnet on QuakeNet, WebChat, Logs) and Skype. Most non-technical discussions happen via Skype.

Future Directions

The main goal is to keep DDNet running, well maintained, and keep a steady stream of new maps from the community.

We submitted DDNet to Steam Greenlight and have been greenlit since then, which means we could release it on Steam now. This would be an opportunity for us to reach more potential players, especially those who have never played Teeworlds before. The main problem right now is that DDNet is a difficult game to get started with, since it requires practice, fine control and patience. Right now work is being done to create a tutorial for new players and generally improve the ease of getting started.

Discussion

You can find discussions about this article on Hacker News, r/programming and also the DDNet Forum.

DDNet Server Statistics with ServerStatus, RRDtool and Nim

2016-05-14T00:00:00-04:00

About a month ago I set up statistics for the official DDNet servers. My motivations for this are:

Monitor the servers more easily
Get notified about server problems
Have nice graphs to look at

The choices for the software used are mainly made to keep resource usage low, a general principle used for DDNet since we run on cheap VPSes all around the world and are limited in CPU and memory resources. In the rest of this post we will explore the 3 major tools used, their purpose in our solution as well as their performance impact:

ServerStatus: Gather live server statistics
RRDtool: Record and graph data
Nim: Favorite programming language for performance and readability

Gathering live server statistics with ServerStatus

We’ve been running BotoX’s ServerStatus to get live server statistics for some time now. It works quite well to quickly notice major server problems like a high load or incoming (D)DoS attacks, provided that you keep an eye out for it.

We use ServerStatus by running its simple Python client on each server to gather interesting information. The client transmits that data by TCP to the C/C++ server, which aggregates it into a JSON file. This JSON file is then fetched and displayed every two seconds by the JavaScript frontend of our Status page.

On a regular Saturday morning the end result looks like this:

On a quick glance you notice that the DDNet.tw server has high CPU usage (which is totally normal since it runs some hefty cron jobs every 20 minutes) and DDNet RUS is receiving a small DoS attack with just 1.6 MB/s (unfortunately also totally normal). Apart from that everything looks fine.

ServerStatus footprint, calculated from Linux /proc statistics as follows (in Nim):

import os, strutils, posix, strfmt
# CPU and memory usage of process, based on PROC(5) and
# http://stackoverflow.com/a/16736599
let
  pid = paramStr(1)
  # Clock ticks per second
  frequency = sysconf(SC_CLK_TCK)
  # Size of memory page in bytes
  pagesize  = sysconf(SC_PAGESIZE)

  uptime    = readFile("/proc/uptime").split[0].parseFloat
  fields    = readFile("/proc/" & pid & "/stat").split

  # Amount of time in user mode
  utime     = fields[13].parseInt
  # Amount of time in kernel mode
  stime     = fields[14].parseInt
  # Amount of children time in user mode
  cutime    = fields[15].parseInt
  # Amount of children time in kernel mode
  cstime    = fields[16].parseInt
  # Time process started after boot
  starttime = fields[21].parseInt

  # Resident Set Size: number of pages in memory
  rssmem    = fields[23].parseInt

  totaltime = utime + stime + cutime + cstime
  seconds   = uptime - (starttime / frequency)

  cpuusage  = 100 * (totaltime / frequency) / seconds
  memusage  = rssmem * pagesize / 1_000_000

echo interp"${cpuusage:.2f} % CPU ${memusage:.2f} MB Memory"

Part	CPU	Memory
Client	0.14 %	3.70 MB
Server (9 clients)	0.58 %	0.18 MB

Recording and graphing data with RRDtool

I haven’t used RRDtool for about 7 years, but it’s still an excellent tool to record data into a fixed-size round robin database. For us three functions of RRDtool are important: create to create the database, update to add a new value to be aggregated into the database, and graph to render the database into a beautiful graph.

CPU, network and memory are the most important resources for me, so their usage should be recorded. Let us use network traffic as an example and create a database:

First we need to think about what data we want to record in the RRD:

sample = 30 seconds

day  =   2880 samples =   6 * 480 pixels, each pixel is 03:00 min
days =  20160 samples =  42 * 480 pixels, each pixel is 21:00 min
days = 141120 samples = 147 * 960 pixels, each pixel is 73:30 min

Then we can use this to create the actual database file:

rrdtool create ddnet.tw-net.rrd `# File name` \
  --step 30 `# Interval in seconds with which data is fed` \
  DS:network_rx:GAUGE:60:0:U `# Data source receiving` \
  DS:network_tx:GAUGE:60:0:U `# DS Sending` \
  RRA:AVERAGE:0.5:6:480 `# Round robin archive for 1 day` \
  RRA:AVERAGE:0.5:42:480 `# RRA for 7 days` \
  RRA:AVERAGE:0.5:147:960 `# RRA for 49 days`

If you’re curious about what exactly happens here, you can find more information in rrdcreate(1).

The resulting ddnet.tw-net.rrd file is just 32 KB in size and will forever stay that exact size. (All our databases together are just 1 MB.) New data in each round robin archive simply overwrites the oldest data. A disadvantage of RRDtool is that you need to think ahead and plan what data you want to store.

The next step is to put new data into our little database, which we should do every 30 seconds:

rrdtool update ddnet.tw-net.rrd N:42:1234

Super simple! 42 is our network_rx value, 1234 the value for network_tx. These values are now aggregated using the AVERAGE and finally put into their respective archives.

Once we have enough values we can finally create the graph, for example for 1 day:

rrdtool graph ddnet.tw-net-1d.png --rigid --base 1000 \
  --width 419 --height 150 --logarithmic --units=si -a PNG \
  `# Calculation over last day only` \
  --vertical-label "Bytes/s" --start now-1d \
  `# Fetch data from RRD file` \
  DEF:network_rx=ddnet.tw-net.rrd:network_rx:AVERAGE \
  DEF:network_tx=ddnet.tw-net.rrd:network_tx:AVERAGE \
  `# Calculate aggregates based on data` \
  VDEF:network_rx_a=network_rx,AVERAGE \
  VDEF:network_rx_m=network_rx,MAXIMUM \
  VDEF:network_rx_c=network_rx,LAST \
  VDEF:network_rx_s=network_rx,TOTAL \
  VDEF:network_tx_a=network_tx,AVERAGE \
  VDEF:network_tx_m=network_tx,MAXIMUM \
  VDEF:network_tx_c=network_tx,LAST \
  VDEF:network_tx_s=network_tx,TOTAL \
  `# Draw area graph in light colors` \
  AREA:network_tx#fee8c8: \
  AREA:network_rx#e0e0e0: \
  `# Draw clear area outline on top` \
  LINE1:network_tx#e34a33:"out" \
  `# Print aggregate values to legend` \
  GPRINT:network_tx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_tx_m:"max\: %6.2lf %sB" \
  GPRINT:network_tx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_tx_s:"sum\: %6.2lf %sB\n" \
  `# Other area outline` \
  LINE1:network_rx#636363:"in " \
  GPRINT:network_rx_a:"avg\: %6.2lf %sB" \
  GPRINT:network_rx_m:"max\: %6.2lf %sB" \
  GPRINT:network_rx_c:"cur\: %6.2lf %sB" \
  GPRINT:network_rx_s:"sum\: %6.2lf %sB\n"

As always, the manual of rrdgraph explains the possibilities.

I’m using RRDtool 1.6.0 instead of 1.4.8 because I very much prefer its density of x-axis labels. Here are the outputs of our database:

RRDtool 1.4.8: RRDtool 1.6.0:

RRDtool footprint:

Part	Runtime	Memory
Create (9 servers)	0.01 s	1.74 MB
Graph (9 servers)	1.99 s	2.82 MB

Putting it together with Nim

To aggregate the raw data from ServerStatus into 30-second packets I use a small Nim program. It automatically creates new databases when a new server is added and keeps them updated:

import common, json, osproc, os, times, strutils, tables

type
  Data = object
    network_rx, network_tx: BiggestInt
    cpu, memory_used, memory_total, swap_used, swap_total: BiggestInt
    load: float

const freq = 30 # report new data to rrd every 30 seconds

var
  lastUpdated: BiggestInt = 0
  dataTable = initTable[string, Data]()
  count = 0
  countTable = initTable[string, int]()

proc rrdCreate(file, dataSources: string) =
  discard execCmd(rrdtool & " create " & file & " --step " & $freq & " " & dataSources &
    " RRA:AVERAGE:0.5:6:480 RRA:AVERAGE:0.5:42:480 RRA:AVERAGE:0.5:147:960")

proc rrdUpdate(file: string, values: varargs[string, `$`]) =
  var valuesString = ""
  for value in values:
    if valuesString.len > 0:
      valuesString.add ":"
    valuesString.add value
  discard execCmd(rrdtool & " update " & file & " N:" & valuesString)

proc updateServer(server: JsonNode) =
  let domain = server["type"].str

  for name, value in dataTable.mgetOrPut(domain, Data()).fieldPairs:
    when value is BiggestInt:
      value += server[name].num
    elif value is float:
      value += server[name].fnum
    else:
      error "Unhandled type in Data object"

  inc countTable.mgetOrPut(domain, 0)

  # Only save data if we got 30 values in the expected time span
  if count == freq and countTable[domain] == freq:
    let data = dataTable[domain]
    if data == Data():
      dataTable.del(domain)
      return
    else:
      dataTable[domain] = Data()

    let
      fileNet = (rrdDir / domain) & "-net.rrd"
      fileCpu = (rrdDir / domain) & "-cpu.rrd"
      fileMem = (rrdDir / domain) & "-mem.rrd"

    if not existsFile fileNet:
      fileNet.rrdCreate("DS:network_rx:GAUGE:60:0:U DS:network_tx:GAUGE:60:0:U")
    if not existsFile fileCpu:
      fileCpu.rrdCreate("DS:cpu:GAUGE:60:0:100 DS:load:GAUGE:60:0:U")
    if not existsFile fileMem:
      filemem.rrdCreate("DS:memory_used:GAUGE:60:0:U DS:memory_total:GAUGE:60:0:U DS:swap_used:GAUGE:60:0:U DS:swap_total:GAUGE:60:0:U")

    fileNet.rrdUpdate(data.network_rx div freq, data.network_tx div freq)
    fileCpu.rrdUpdate(min(data.cpu div freq, 100), data.load / freq)
    fileMem.rrdUpdate(data.memory_used div freq, data.memory_total div freq, data.swap_used div freq, data.swap_total div freq)

proc updateAllServers =
  let statsJson = parseFile statsJsonFile
  let newUpdated = parseBiggestInt statsJson["updated"].str

  if newUpdated <= lastUpdated:
    return

  inc count

  for server in statsJson["servers"]:
    try:
      updateServer(server)
    except:
      discard

  if count == freq:
    count = 0
    for val in countTable.mvalues:
      val = 0

while true:
  let startTime = epochTime()

  updateAllServers()

  sleep(int(epochTime() - startTime + 1) * 1000) # every second

The final graphs can be seen on the DDNet Server Statistics page.

Nim monitor footprint:

Part	CPU	Memory
Monitor (9 servers)	0.03 %	0.89 MB

Alerts through Cron and Mail

Now we certainly have nice graphs, but automated alerts about suspicious events would be even better, for example:

Network traffic over 2 MB/s for 4 min
Memory and swap over 90% for 4 min
CPU over 90% for 21 min
Load over 10 for 21 min
Server unreachable for 1 hour

To check these conditions a cron job is run regularly and thanks to a MAILTO entry a mail is sent when an alert has been triggered.

We can get out a single value from the database to standard output using PRINT:

rrdtool graph x -s-4min \
  DEF:v=ddnet.tw-net.rrd:network_rx:AVERAGE \
  VDEF:vm=v,AVERAGE \
  PRINT:vm:%lf

The alert program itself is written in Nim as well and merely gets a few values from the databases and checks if the limits are exceeded:

import common, osproc, os, json, strutils

proc get(file, value, time: string): float =
  let (output, errorCode) = execCmdEx(rrdtool & " graph x -s -" & time & " DEF:v=" & file & ":" & value & ":AVERAGE VDEF:vm=v,AVERAGE PRINT:vm:%lf")

  if errorCode != 0:
    raise newException(ValueError, "Error code " & $errorCode & " from rrdtool: " & output)

  result = output.splitLines[^2].parseFloat

if paramCount() != 1:
  echo "alert [1d|7d|49d]"
  quit 1

let statsJson = parseFile statsJsonFile
for server in statsJson["servers"]:
  let
    domain = server["type"].str
    name = server["name"].str
    fileNet = (rrdDir / domain) & "-net.rrd"
    fileCpu = (rrdDir / domain) & "-cpu.rrd"
    fileMem = (rrdDir / domain) & "-mem.rrd"

  template alert(s: string) = echo name, " ", s

  case paramStr(1)
  of "1d":
    if fileNet.get("network_rx", "4min") + fileNet.get("network_tx", "4min") > 2_000_000:
      alert "network traffic over 2 MB/s for 4 min"

    if fileMem.get("memory_used", "4min") + fileMem.get("swap_used", "4min") > 0.9 * (fileMem.get("memory_total", "4min") + fileMem.get("swap_total", "4min")):
      alert "memory and swap over 90% for 4 min"

  of "7d":
    if fileCpu.get("cpu", "21min") > 90.0:
      alert "CPU over 90% for 21 min"
    if fileCpu.get("load", "21min") > 10.0:
      alert "Load over 10 for 21 min"

  of "49d":
    let network_rx = fileNet.get("network_rx", "4410")
    if network_rx != network_rx: # NaN
      alert "unreachable for 1 hour"

  else:
    echo "unknown parameter ", paramStr(1)
    quit 1

Nim alert footprint:

Part	Runtime	Memory
Alert (9 servers)	0.25 s	2.12 MB

Conclusion

Live view of the last day of the DDNet.tw web server, also hosting this blog:

You can see the full graphs on the DDNet Server Statistics page. As usual you can find the entire source code in our git repository.

All in all the system runs on very little resources, puts out some nice graphs and alerts me automatically about defined problems.

Part	Runtime	CPU	Memory
Client		0.14 %	3.70 MB
Server (9 clients)		0.58 %	0.18 MB
Create (9 servers)	0.01 s		1.74 MB
Graph (9 servers)	1.99 s		2.82 MB
Monitor (9 servers)		0.03 %	0.89 MB
Alert (9 servers)	0.25 s		2.12 MB

DDNet Live: Twitch spectates an online game

2016-01-10T00:00:00-05:00

Last night I had an idea and implemented it, soo let’s see what will happen. But first, the idea:

Have a livestream [1] of DDNet running non-stop [2] that always shows some interesting [3] players on the server.

The resulting livestream is running on Twitch. All the scripts are on Github

1. Livestream

It’s surprisingly simple to livestream from Linux to Twitch. Only FFmpeg is needed:

#!/bin/sh

INRES="1280x720" # input resolution
FPS="30" # target FPS
GOP="60" # i-frame interval, should be double of FPS, 
GOPMIN="30" # min i-frame interval, should be equal to fps, 
THREADS="2"
CBR="2000k" # constant bitrate (should be between 1000k - 3000k)
QUALITY="ultrafast"  # one of the many FFMPEG preset
AUDIO_SRATE="44100"
AUDIO_CHANNELS="2" # 1 for mono output, 2 for stereo
AUDIO_ERATE="96k" # audio encoding rate
SERVER="live-fra" #  http://bashtech.net/twitch/ingest.php for list
source ./secret.sh

ffmpeg -v 0 -f x11grab -s "$INRES" -r "$FPS" -i :0.0 \
  -f alsa -i pulse -f flv -ac $AUDIO_CHANNELS \
  -b:a $AUDIO_ERATE -ar $AUDIO_SRATE \
  -vcodec libx264 -g $GOP -keyint_min $GOPMIN -b:v $CBR \
  -minrate $CBR -maxrate $CBR -vf "format=yuv420p"\
  -preset $QUALITY -acodec libmp3lame -threads $THREADS \
  -strict normal -bufsize $CBR \
  "rtmp://$SERVER.twitch.tv/app/$STREAM_KEY"

This works pretty well, but takes quite a bit of CPU on my old computer. So instead I wanted to run it on my new server with a Haswell-era J1900 CPU.

Possible ways to improve performance:

Use an OpenGL recorder instead of x11grab
Get Intel Quick Sync Video working for hardware H264 encoding

2. Running Non-Stop

My server is a small and cheap ASRock Q1900-ITX:

The nice part is that it’s passively cooled and quite power efficient, drawing only 5 Watts in idle. This machine has been running as my home server for quite some time, but barely gets any action. Let’s change that!

The X server starts without problems even without any monitors attached, the only thing that’s left to do is increasing the framebuffer size so that our game can run in it:

$ cat ddnet.sh
#!/bin/sh

pulseaudio --start
xrandr --fb 1280x720 # Adjust framebuffer
cp settings_ddnet.cfg ~/.teeworlds/ # Restore backup
cd ddnet && ./DDNet

$ xinit ddnet.sh

We don’t even need a window manager. After all we just run a single window in exactly the resolution of the framebuffer.

The FFmpeg recording still works in exactly the same way. Unfortunately I couldn’t get H264 hardware encoding to work with the J1900 CPU. Related bug reports make me believe it just doesn’t work on these cheaper Intel CPUs.

3. Artificial Intelligence Twitch Control

Now that we have the game running and are streaming it to Twitch, we need to control it somehow. My goal was to find an approach that always shows some interesting players in action, so that you could watch the stream all day and enjoy it.

But finding a reasonable way to do this seems too complicated and I didn’t look forward to hacking the DDNet C/C++ source code, so instead I opted to utilize an existing system in DDNet: FIFO command input!

DDNet servers and clients can be remote controlled through a FIFO file. This is very useful to send the same commands to dozens of servers at once. But for the client its use was pretty limited, until now!

Instead of thinking of an algorithm to find interesting players, why not let the Twitch viewers themselves control who they want to watch through the chat?

I did not modify the DDNet source code in any way and instead wrote a small Nim script based on the IRC module to connect to Twitch’s IRC server and forward the commands to the FIFO:

import irc, strutils, secret

const forbiddenCommands = @["exec", "quit", "exit", "disconnect"]

var
  client = newIrc("irc.twitch.tv", nick = "ddnetlive",
    serverPass = serverPass, joinChans = @["#ddnetlive"])
  fifo = open("input.fifo", fmWrite)

proc handle(nick, cmd: string) =
  for f in forbiddenCommands:
    if cmd.contains(f):
      return

  echo nick, ": ", cmd
  stdout.flushFile()

  fifo.write(cmd)
  fifo.write("\n")
  fifo.flushFile()

client.connect()
var event: TIrcEvent

while true:
  if client.poll(event):
    case event.typ
    of EvConnected:
      discard
    of EvDisconnected, EvTimeout:
      client.reconnect()
    of EvMsg:
      case event.cmd
      of MPrivMsg:
        handle(event.nick, event.params[^1])
      else:
        discard

Right now there is really not much limitation to what you can do. Only a handful of commands are explicitly blocked, otherwise the client can be controlled freely and every single chat message is sent to the client. The client commands and client settings list the available commands and settings. Chat commands can be sent to the server as well through the say command. Some examples:

connect ger.ddnet.org:8303
team -1 # Join spectators
spectate_next # Spectate the next player
spectate 0 # Spectate player with ID 0
say Hi from twitch.tv/ddnetlive # Write chat messages
player_name DDNetLive # Change name

Let’s see how this goes. Luckily the Nim script runs independently from the server, so I will be able to make changes to it on the fly. Just be nice and don’t cause any trouble, thanks.

Feel free to head over to Twitch, watch the action and control the server using the Twitch chat. Twitch has some delay, so it takes a few seconds for you to see your command executed.