HookRace Blog2023-09-07T18:29:49+02:00https://hookrace.net/Dennis Felsingdennis@felsing.orgHaskell: Game Programming with GIF Streams2023-09-07T00:00:00+02:00https://hookrace.net/blog/haskell-game-programming-with-gif-streams
<p>Note: This is translated from the <a href="https://github.com/def-/gifstream/blob/master/Aufgabe.pdf">German original</a>, which was used as a homework for the Programming Paradigms course at Karlsruhe Institute of Technology a long long time ago when I was co-holding the practical courses.
<!--more--></p>
<p>Snake is a computer game in which a snake has to be moved through a playing field.
Eating food increases the snake’s length.
When the snake collides with a wall or itself the game ends.</p>
<p><a href="/public/snake.gif"><img src="/public/snake.gif" alt="snake.gif" /></a></p>
<p>In this homework you will implement Snake in Haskell.
For this you will require the framework from the <a href="https://github.com/def-/gifstream">gifstream repository</a>.</p>
<p>The output of the game happens in an animated GIF stream, which you can watch in your browser.
64 colors are supported, which can be respresented as Int tuples of <code class="language-plaintext highlighter-rouge">(0,0,0)</code> up to <code class="language-plaintext highlighter-rouge">(3,3,3)</code>.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">type</span> <span class="kt">RGB</span> <span class="o">=</span> <span class="p">(</span><span class="kt">Int</span><span class="p">,</span><span class="kt">Int</span><span class="p">,</span><span class="kt">Int</span><span class="p">)</span></code></pre></figure>
<p>A single frame of a GIF is defined as a list of rows, wherein each row is a list of RGB values.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">type</span> <span class="kt">Frame</span> <span class="o">=</span> <span class="p">[[</span><span class="kt">RGB</span><span class="p">]]</span></code></pre></figure>
<p>The framework provides a \texttt{server} function, which runs an HTTP server on the supplied port.
The server sends each client a new frame of the GIF animation at the set interval.
In the passed logic function new frames will be generated dynamically.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">server</span> <span class="o">::</span> <span class="kt">PortNumber</span> <span class="o">-></span> <span class="kt">Int</span> <span class="o">-></span> <span class="kt">Logic</span> <span class="o">-></span> <span class="kt">IO</span> <span class="nb">()</span></code></pre></figure>
<p>The <a href="https://github.com/def-/gifstream/blob/master/Snake.hs">Snake.hs</a> file contains the basis for writing the Snake game.
Compile the game and run it (You’ll need to install the <code class="language-plaintext highlighter-rouge">network</code> and <code class="language-plaintext highlighter-rouge">random</code> Haskell packages too):</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>ghc <span class="nt">-O3</span> <span class="nt">-threaded</span> Snake.hs
<span class="o">[</span>1 of 2] Compiling GifStream <span class="o">(</span> GifStream.hs, GifStream.o <span class="o">)</span>
<span class="o">[</span>2 of 2] Compiling Main <span class="o">(</span> Snake.hs, Snake.o <span class="o">)</span>
Linking Snake ...
<span class="nv">$ </span>./Snake
Listening on http://127.0.0.1:5002/</code></pre></figure>
<p>Open the supplied address in a browser.
By pressing the WASD keys in the terminal you can influence the GIF in your browser.</p>
<p>Other participants in your network can watch the GIF stream as well, by using your network IP address instead of <code class="language-plaintext highlighter-rouge">127.0.0.1</code>.</p>
<p>Furthermore it is possible to record the GIF stream and watch it later:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">wget <span class="nt">-O</span> game.gif http://127.0.0.1:5002/</code></pre></figure>
<p>The most important function in <a href="https://github.com/def-/gifstream/blob/master/Snake.hs">Snake.hs</a> is <code class="language-plaintext highlighter-rouge">logic</code>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">logic</span> <span class="n">wait</span> <span class="n">getInput</span> <span class="n">sendFrame</span> <span class="o">=</span> <span class="n">initialState</span> <span class="o">>>=</span> <span class="n">go</span>
<span class="kr">where</span>
<span class="n">go</span> <span class="p">(</span><span class="kt">State</span> <span class="n">oldAction</span> <span class="n">snake</span> <span class="n">food</span><span class="p">)</span> <span class="o">=</span> <span class="kr">do</span>
<span class="n">input</span> <span class="o"><-</span> <span class="n">getInput</span>
<span class="c1">-- Generate new state</span>
<span class="kr">let</span> <span class="n">action</span> <span class="o">=</span> <span class="n">charToAction</span> <span class="n">input</span> <span class="n">oldAction</span>
<span class="kr">let</span> <span class="n">newSnake</span> <span class="o">=</span> <span class="n">snake</span>
<span class="kr">let</span> <span class="n">newFood</span> <span class="o">=</span> <span class="n">food</span>
<span class="kr">let</span> <span class="n">frame</span> <span class="o">=</span> <span class="kr">case</span> <span class="n">action</span> <span class="kr">of</span>
<span class="kt">MoveUp</span> <span class="o">-></span> <span class="n">replicate</span> <span class="n">height</span> <span class="p">(</span><span class="n">replicate</span> <span class="n">width</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">))</span>
<span class="kt">MoveDown</span> <span class="o">-></span> <span class="n">replicate</span> <span class="n">height</span> <span class="p">(</span><span class="n">replicate</span> <span class="n">width</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">0</span><span class="p">))</span>
<span class="kt">MoveLeft</span> <span class="o">-></span> <span class="n">replicate</span> <span class="n">height</span> <span class="p">(</span><span class="n">replicate</span> <span class="n">width</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="kt">MoveRight</span> <span class="o">-></span> <span class="n">replicate</span> <span class="n">height</span> <span class="p">(</span><span class="n">replicate</span> <span class="n">width</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="n">sendFrame</span> <span class="p">(</span><span class="n">scale</span> <span class="n">zoom</span> <span class="n">frame</span><span class="p">)</span>
<span class="n">wait</span>
<span class="n">go</span> <span class="p">(</span><span class="kt">State</span> <span class="n">action</span> <span class="n">newSnake</span> <span class="n">newFood</span><span class="p">)</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">logic</code> function creates an initial state for the Snake game and passes it to the <code class="language-plaintext highlighter-rouge">go</code> function.
This function in turn uses <code class="language-plaintext highlighter-rouge">getInput</code> to read the key which was pressed last.
Afterwards a new game state is generated.
The shown frame is chosen based on the pressed key.
Finally <code class="language-plaintext highlighter-rouge">sendFrame</code> send the new frame to all connected clients.
As part of this function each frame is scaled by the <code class="language-plaintext highlighter-rouge">scale</code>.
The call to <code class="language-plaintext highlighter-rouge">wait</code> causes waiting for the set time <code class="language-plaintext highlighter-rouge">delay</code>, which defaults to 100 ms.
At the end of the function it calls itself tail recursively with the newly generated state.
The goal of this task is to extend the game logic in <code class="language-plaintext highlighter-rouge">logic</code> step by step so that you can play Snake in the end.</p>
<h2 id="printing-the-playing-field">Printing the Playing Field</h2>
<p>Use the current state to generate an image and print this instead of the simple single-color image.</p>
<p>Write a list <code class="language-plaintext highlighter-rouge">fieldPositions</code> which saves the coordinates of the playing field in its corresponding position.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">fieldPositions</span> <span class="o">::</span> <span class="p">[[</span><span class="kt">Position</span><span class="p">]]</span></code></pre></figure>
<p>The size of the field is saved in <code class="language-plaintext highlighter-rouge">width</code> and <code class="language-plaintext highlighter-rouge">height</code>.
For a field of the size 3x4 <code class="language-plaintext highlighter-rouge">fieldPositions</code> would look as follows:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">fieldPositions</span> <span class="o">=</span> <span class="p">[[(</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">),(</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">),(</span><span class="mi">2</span><span class="p">,</span><span class="mi">0</span><span class="p">)]</span>
<span class="p">,[(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">),(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">),(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)]</span>
<span class="p">,[(</span><span class="mi">0</span><span class="p">,</span><span class="mi">2</span><span class="p">),(</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">),(</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">)]</span>
<span class="p">,[(</span><span class="mi">0</span><span class="p">,</span><span class="mi">3</span><span class="p">),(</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="p">),(</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">)]]</span></code></pre></figure>
<p>Implement a <code class="language-plaintext highlighter-rouge">colorize</code> function which maps a single position of the image to a color so that the new frame can be created through <code class="language-plaintext highlighter-rouge">let frame = map (map (colorize newSnake newFood)) fieldPositions</code>.
A field should be colored differently depending on whether this position is part of the snake, food or background.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">colorize</span> <span class="o">::</span> <span class="p">[</span><span class="kt">Position</span><span class="p">]</span> <span class="o">-></span> <span class="kt">Position</span> <span class="o">-></span> <span class="kt">Position</span> <span class="o">-></span> <span class="kt">RGB</span></code></pre></figure>
<h2 id="snake-behavior">Snake Behavior</h2>
<p>Next implement state changes so that the game logic can be written as <code class="language-plaintext highlighter-rouge">let newSnake = moveSnake snake food action</code>.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">moveSnake</span> <span class="o">::</span> <span class="p">[</span><span class="kt">Position</span><span class="p">]</span> <span class="o">-></span> <span class="kt">Position</span> <span class="o">-></span> <span class="kt">Action</span> <span class="o">-></span> <span class="kt">Position</span></code></pre></figure>
<p>A snake is defined as a list of positions.
The new snake receives a new head depending on the passed action.
<code class="language-plaintext highlighter-rouge">Action</code> is defined as follows:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="kr">data</span> <span class="kt">Action</span> <span class="o">=</span> <span class="kt">MoveLeft</span> <span class="o">|</span> <span class="kt">MoveRight</span> <span class="o">|</span> <span class="kt">MoveUp</span> <span class="o">|</span> <span class="kt">MoveDown</span> <span class="kr">deriving</span> <span class="kt">Eq</span></code></pre></figure>
<p>At the tail the last element is cut off, except when the snake just reached food.</p>
<p>It has to be ensured that the user-chosen action is even possible.
Write a function <code class="language-plaintext highlighter-rouge">validateAction</code> so that the game logic can be extended by <code class="language-plaintext highlighter-rouge">let action = validateAction oldAction (charToAction input oldAction)</code>.
For this <code class="language-plaintext highlighter-rouge">validateAction</code> shall only return a new action when this is possible.
Otherwise the old action shall be returned.</p>
<p><a href="/public/listmonster.png"><img src="/public/listmonster.png" alt="listmonster.png" /></a>
Picture from <a href="http://www.learnyouahaskell.com/">Learn You A Haskell</a></p>
<h2 id="food-behavior">Food Behavior</h2>
<p>Next implement the state change of the food so that the game logic can be extended by <code class="language-plaintext highlighter-rouge">newFood <- moveFood newSnake food</code>.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">moveFood</span> <span class="o">::</span> <span class="p">[</span><span class="kt">Position</span><span class="p">]</span> <span class="o">-></span> <span class="kt">Position</span> <span class="o">-></span> <span class="kt">IO</span> <span class="kt">Position</span></code></pre></figure>
<p>When the snake doesn’t currently eat the food the old position of the food can be returned directly.
Otherwise the new position of the food shall be chosen.
Avoid that the food appears inside of the body of the snake</p>
<p>Random numbers between x and y (inclusively) can be generated in the <code class="language-plaintext highlighter-rouge">do</code> syntax using <code class="language-plaintext highlighter-rouge">r <- randomRIO (x,y)</code>.
Thus import the <code class="language-plaintext highlighter-rouge">System.Random</code> module.</p>
<h2 id="end-of-the-game">End of the Game</h2>
<p>Adapt the end of <code class="language-plaintext highlighter-rouge">logic</code> so that with <code class="language-plaintext highlighter-rouge">newSnake</code> the validity of the new state is checked.
In an invalid state the game shall be restarted by calling <code class="language-plaintext highlighter-rouge">initialstate >>= go</code>.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">checkGameOver</span> <span class="o">::</span> <span class="p">[</span><span class="kt">Position</span><span class="p">]</span> <span class="o">-></span> <span class="kt">Bool</span></code></pre></figure>
<p>A full solution is available in the <a href="https://github.com/def-/gifstream/blob/master/SnakeFinished.hs">GitHub repository</a>.</p>
<h2 id="bonus">Bonus</h2>
<p>Program another game with GIF stream output, for example Pong, Tetris or Conway’s Game of Life.</p>
<p>Related: <a href="/blog/time.gif/">time.gif</a></p>
God writes Haskell2023-06-03T00:00:00+02:00https://hookrace.net/blog/god-writes-haskell
<p>God famously does not play dice with the universe, but he seems to enjoy writing <a href="https://www.haskell.org/">Haskell</a>:
<!--more--></p>
<ol>
<li>
<p>Consider the <a href="https://en.wikipedia.org/wiki/Wave%E2%80%93particle_duality">wave-particle duality</a> in quantum mechanics. Every particle behaves as a wave, as long as you haven’t interacted with it. Thanks to Haskell’s <a href="https://wiki.haskell.org/Lazy_evaluation">lazy evaluation</a> values are also only evaluated once they are accessed (interacted with particles), and stay unevaluated thunks (waves) in the meantime.</p>
</li>
<li>
<p>Two particles can be <a href="https://en.wikipedia.org/wiki/Quantum_entanglement">Quantum-entangled</a>, so that their states depend on each other, even though the particles are seperated by any distance. In Haskell a value, whether it’s evaluated yet or not, can also be shared and then used in a totally different location in the program without having to copy it. The value is even immutable, so that you can’t change it from one location and thus influence the other. Similarly for entangled particles you can’t manipulate one to change the state of the other particle, which might be far away and thus break the maximum <a href="https://en.wikipedia.org/wiki/Speed_of_light">speed of information</a>.</p>
</li>
<li>
<p>Since values are immutable they have to be cleaned up more often in Haskell than typically in imperative languages. <a href="https://www.haskell.org/ghc/">GHC</a>, the most commonly used Haskell compiler, allocates new data in a special area. Only after a supernova will the still-relevant data be ejected into the larger universe.</p>
</li>
<li>
<p>Haskell beginners often use lists instead of arrays. You can’t do random access in a linked list, but only access the first element and then the rest of the list. The real world also doesn’t allow you random access, you are limited by the speed of light and have to go from one location to the next.</p>
</li>
<li>
<p>Time also seems to be a linked list, not even doubly linked, since you can’t go back after accessing the current element. Seems like an awkward bug.</p>
</li>
<li>
<p>Since the Haskell type system is so good at catching bugs, you often feel like you don’t even need to write tests. This is unfortunately untrue, as the strange physical bugs of our universe demonstrate: The speed of light happens to stay the same, no matter what speed you move at.</p>
</li>
<li>
<p>There seems to be a long-term memory leak in Haskell, which is kind of easy to happen since you can have unevaluated thunks growing in size as long as they are still reachable. Unfortunately this memory leak will keep growing until it consumes the entire universe, which will then die of its <a href="https://en.wikipedia.org/wiki/Heat_death_of_the_universe">heat death</a>.</p>
</li>
</ol>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=37414624">Hacker News</a></p>
Database Testing at Yugabyte2023-01-14T00:00:00+01:00https://hookrace.net/blog/database-testing-at-yugabyte
<p><a href="https://www.yugabyte.com/yugabytedb/">YugabyteDB</a> is a cloud-native database for business-critical enterprise applications. It is designed to provide continuous availability as well as horizontal scalability, while retaining a strong set of RDBMS features. This objective creates a strong quality incentive for us in the Yugabyte Quality Assurance (QA) team. As a member of this team, I am giving an overview of the testing philosophy, approaches, and implementations for YugabyteDB.</p>
<p>Read the rest of the blog post over on the <a href="https://www.yugabyte.com/blog/yugabytedb-database-testing/">YugabyteDB blog</a>.</p>
€9 Ticket2022-08-20T00:00:00+02:00https://hookrace.net/blog/euro9-ticket
<p>As a relief for rising energy costs and inflation Germany is offering nearly-free public transportation for the months of June, July and August. For just €9 per month you can take any kind of regional public transportation in all of Germany. That’s basically everything except the IC/ICE for long distance. This means that a monthly ticket is now as cheap as what I would normally pay for a single route. In this post I want to talk about my experiences with this <a href="https://www.bahn.com/en/offers/regional/9-euro-ticket-en">€9 ticket</a>.</p>
<p>Update 2022-09-01: The €9 ticket ended yesterday without a similar replacement system in place. So many people are again back to paying about €100 per month for much smaller regional tickets instead (see for example this <a href="https://old.reddit.com/r/de/comments/x2x01s/70_euro_f%C3%BCr_4_stationen_endlich_zur%C3%BCck_in_meinem/">German Reddit thread</a>. Financially it makes more sense for me personally to go back to driving. (I hope the gasoline in the car is still good after three months.) My hope is that these three months will stay in people’s minds in Germany and shape the future of public transportation to be cheaper and simpler:</p>
<!--more-->
<p><a href="/public/9euro/fahrkartenautomat.webp"><img src="/public/9euro/fahrkartenautomat.webp" alt="Fahrkartenautomat" /></a>
Image via <a href="https://old.reddit.com/r/de/comments/26rty0/ausland_vs_deutschland/">Reddit</a></p>
<h2 id="train-stations">Train Stations</h2>
<p><a href="https://assets.static-bahn.de/dam/jcr:f55563b4-0ccf-4669-bb92-459a5988691c/20220221_LN_S-Bahn_RN_867x385_Mireo.pdf"><img style="width: 100vw; max-width: 100vw; margin-left: -50vw; position: relative; left: 50%" alt="VRN Plan" src="/public/9euro/vrn.png" /></a></p>
<p>In the above graphic you can see the train stations in our area. Since we are situated in the <a href="https://en.wikipedia.org/wiki/Rhine-Neckar">Rhine-Neckar metropolitan region</a> the railway network is pretty good here.</p>
<iframe class="osm-map" allowfullscreen="" src="//umap.openstreetmap.fr/en/map/train-stations_798502?scaleControl=true&miniMap=false&scrollWheelZoom=false&zoomControl=true&allowEdit=false&moreControl=false&searchControl=null&tilelayersControl=null&embedControl=null&datalayersControl=true&onLoadPanel=undefined&captionBar=false"></iframe>
<p>(map made with <a href="https://umap.openstreetmap.fr/en/">umap</a>, routes made with <a href="https://www.graphhopper.com/">GraphHopper</a>)</p>
<p>The closest train stop from us is St. Ilgen-Sandhausen at 2 km distance. It offers direct connections to the cities of Heidelberg (160k inhabitants), Mannheim (309k), Karlsruhe (313k), Darmstadt (159k) and Frankfurt (753k). For other directions I prefer going to a different train station since switching times are either too long or there is too much risk of missing the connecting train. Only 89% of regional trains are <a href="https://www.deutschebahn.com/de/konzern/konzernprofil/zahlen_fakten/puenktlichkeitswerte-6878476">on time</a> in Germany, with some routes and stations having much worse rates. This is a far cry from Switzerland’s <a href="https://www.admin.ch/gov/de/start/dokumentation/medienmitteilungen.msg-id-83485.html">95% punctuality</a> with an even stricter definition of 3 minutes versus 5 minutes delay in Germany.</p>
<p>Another closeby train station we commonly use is HD-Weststadt/Südstadt at 6 km distance, which is good to cycle and can also be reached via Tram. Occasionally I used Heidelberg Hauptbahnhof as well as Wiesloch-Walldorf, both at slightly under 10 km distance, which are a bit larger and thus are used as stops for Regional-Express trains. These go faster than the Regionalbahn and S-Bahn trains and thus also have fewer stops.</p>
<h2 id="cycling-trips">Cycling Trips</h2>
<iframe class="osm-map" allowfullscreen="" src="//umap.openstreetmap.fr/en/map/cycling-and-jogging-routes_798422?scaleControl=true&miniMap=false&scrollWheelZoom=false&zoomControl=true&allowEdit=false&moreControl=false&searchControl=null&tilelayersControl=null&embedControl=null&datalayersControl=true&onLoadPanel=undefined&captionBar=false"></iframe>
<p>The above map contains some of the cycling trips I took by combining them with the €9 ticket. (map made with <a href="https://umap.openstreetmap.fr/en/">umap</a>, routes made with <a href="https://www.graphhopper.com/">GraphHopper</a>)</p>
<p>Last time I posted about cycling and work was about <a href="/blog/cycling-to-work/">commuting by bicycle for a year</a>. I kept this up and was super happy with this way of getting to <a href="https://www.sap.com/">SAP</a>’s headquarter in Walldorf and the included exercise. After a while I moved together with my wife and my bike commute lengthened to 10 km each direction, which was still perfectly fine. Then Covid forced me in March of 2020 to work from home I switched to regular cycling trips in the afternoon since I started working early in the morning, in order to have meetings with colleagues in Korea and China. Two 50 km cycling trips per week got me to the same level as the daily commute used to do, and I have a nice route I regularly take, which covers a lot of forest and lake.</p>
<p>In January I switched to a new job at <a href="https://www.yugabyte.com/">Yugabyte</a>. As the database we develop is distributed, so are the people I work with. Since most meetings fall into US times, I am now working more during the afternoon and evening. This means my cycling trips moved to the morning instead.</p>
<p>Previously I had a job ticket for three years, but other than the €9 ticket it limits the range I can take public transportation in, so that many of my destinations would be barely outside of the covered range. When looking at a map of Germany with all the different transport associations, each with their own tickets, prices and rules, I feel reminded of <a href="https://de.m.wikipedia.org/wiki/Datei:HRR_1789.png">historical maps of the Holy Roman Empire</a>:</p>
<p><a href="/public/9euro/verbuende.png"><img src="/public/9euro/verbuende.png" alt="Public Transport Associations in Germany" /></a></p>
<p>Good luck figuring out what ticket to get to combine with your job ticket. Luckily the €9 ticket changed this, at least temporarily, making life simpler.</p>
<p><a href="/public/9euro/oldtown.jpg"><img src="/public/9euro/oldtown.jpg" alt="Old town at Neckar" /></a></p>
<p>I used the ticket during the working days to discover new cycling routes. By cycling one direction and taking a direct train connection back, I can cover nearly twice the distance as usual with the same starting and stopping point. With this method I took many routes I had never taken before, discovering beautiful cycling paths along the Neckar river for example. Luckily the trains are not too crowded during working days, especially outside of rush hour. The cycling usually takes around three times as long as the train ride back.</p>
<p><a href="/public/9euro/empty.jpg"><img src="/public/9euro/empty_small.jpg" alt="Empty train" /></a></p>
<p>The image above shows an almost empty train during a working day.</p>
<p>Even with a single train connection things can go wrong though. One time the train in front of ours started burning and so we got delayed for an unknown time. But since I had my bicycle with me, that meant I could just cycle back the rest of the way and get some extra exercise for free. If I was really out of energy I could have waited in the train, but often busses are used as a replacement, and bicycles on busses don’t fit well.</p>
<p>The weather was unusually dry the last months, the last few days had the first few drops of rain I remember since the beginning of June. This is of course bad for nature here, as I saw at the Rhine river, which is much lower than usual. For cycling it’s quite good though, only got wet once.</p>
<h2 id="jogging">Jogging</h2>
<p><a href="/public/9euro/forest.jpg"><img src="/public/9euro/forest.jpg" alt=""Kleiner Odenwald" forest" /></a></p>
<p>This amount of cycling of course meant that my bicycle had to get some repairs done. While I had it at the repair shop, I tried out jogging from Heidelberg back to my home town Leimen. The route goes through the “Kleiner Odenwald” forest for nearly the entire way, and the rest are some fields. It goes through Heidelberg’s highest mountain, the Königstuhl, with a nice view of the city and surrounding area.</p>
<p><a href="/public/9euro/koenigstuhl.jpg"><img src="/public/9euro/koenigstuhl_small.jpg" alt="View from Königstuhl" /></a></p>
<p>The “Ladder to Heaven” in the beginning with its 1300 steps accounts for most of the elevation and is still too tough for me to do at faster than walking speed. After I got my bicycle back I still kept doing this jogging route a few times per week.</p>
<p><img src="/public/9euro/jogging.png" alt="Elevation for Jogging" /></p>
<h2 id="nearby-excursions">Nearby Excursions</h2>
<p>There are lots of cities and nice nature nearby. So we used public transportation for excursions on every weekend. It was noticable that the trains were crowded, and at times people couldn’t even enter the train anymore. When going to the cinema we noticed that we should really catch the last train at 22:30, since the next connection would take all night.</p>
<p><a href="/public/9euro/excursion.jpg"><img src="/public/9euro/excursion_small.jpg" alt="Cinema excursion" /></a></p>
<p>One time the train back from Frankfurt was late, then got cancelled. The next train was also delayed, and now overcrowded because it had twice the passengers. Unfortunately it took me too long to realize that with more than 20 minutes delay, we should be able to take high speed ICE trains as a replacement and get the cost reimbursed by Deutsche Bahn.</p>
<p>You still need to wear masks as a Covid prevention on public transportation in Germany. This is basically the only remaining occasion, other than at the Doctor’s office, where you have to wear masks. Even with all the travel using public transportation and basically no distancing we haven’t caught Covid as far as we are aware based on symptoms and some rapid tests.</p>
<h2 id="eating-out">Eating Out</h2>
<p><a href="/public/9euro/food1.jpg"><img src="/public/9euro/food1_small.jpg" alt="Food" /></a></p>
<p>My wife picked out many great restaurants in nearby cities for us to try out. By car it would not make sense to just drive to another city for a dinner, but using public transportation we had no problem spending an hour on a train, where you can comfortable talk or surf the web, to reach our destinations. This probably has a really good economic effect on restaurants and cafes, many were so fully packed that we got the last seats or had to reserve beforehand.</p>
<p><a href="/public/9euro/food2.jpg"><img src="/public/9euro/food2_small.jpg" alt="More food" /></a></p>
<h2 id="family-visits">Family Visits</h2>
<p>For trips to cities and cycling trips we mostly take direct train connections. That usually works quite well. To visit family however the destinations are smaller villages, so the connections are worse and there is no direct connection at all. So a 40 minute drive turns into a 2 hour train ride with 3 different trains instead. Coincidentally this is the same time I need for the route by bicycle.</p>
<p>There is a relatively high chance of missing a connecting train, or of getting stuck in the middle of the route. One time a truck crashed into the train bridge, so our train couldn’t continue. Instead of waiting around for an hour or two we took the train back home and, for the only time during these three months so far, had to switch to the car to keep our timeline. At least the highway was quite empty, there seems to be some conflicting data on whether the €9 ticket actually has an effect on reducing car traffic though.</p>
<h2 id="getting-visits">Getting Visits</h2>
<p>While my wife and I didn’t do any long distance travel using the €9 ticket, others did. I had a co-developer of <a href="https://ddnet.org/">DDraceNetwork</a> visit all the way from Berlin using the €9 ticket. For people who have time it has been much easier to visit others with the €9 ticket, without having to worry about cost.</p>
<h2 id="while-on-vacation">While on Vacation</h2>
<p>While we didn’t use the €9 ticket to go to a vacation, it was still useful during a car-based vacation. While driving to Berchtesgadener Land we saw a train line that seemed to lead to Berchtesgaden. So we stopped at the closeby train station and spontaneously took the train instead, which lead us through beautiful sceneries.</p>
<p><a href="/public/9euro/berchtesgaden.jpg"><img src="/public/9euro/berchtesgaden.jpg" alt="Berchtesgaden" /></a></p>
<p>On the way back from the same vacation we had a stop in Munich. Instead of driving the car all the way into the city center I checked for the closest free Park and Ride garage from the highway. After parking the car there, the train was unfortunately cancelled, so we had to take a bus and metro, but still reached the city center quickly.</p>
<p><a href="/public/9euro/foreign1.jpg"><img src="/public/9euro/foreign1_small.jpg" alt="Public transportation on Mallorca" /></a></p>
<p>After having used public transportation a lot back at home, we also felt more comfortable using it on vacation in other countries, even if it costs a bit more there. So on Mallorca we explored the entire island using the extensive bus network instead of renting a car. The photo above shows the most distant bus stop we reached, at a lighthouse at the end of a long curvy road. Next week we are taking the TGV to Paris, which tops out at 320 km/h and manages the 550 km route in just 2:36, an average of 212 km/h. The image below is luckily not the TGV:</p>
<p><a href="/public/9euro/foreign2.jpg"><img src="/public/9euro/foreign2_small.jpg" alt="More public transportation on Mallorca" /></a></p>
<p>Going to the airport by train worked fine, but we had to get there a few hours earlier in case was some train delay causing us to miss the connecting train. On the way back there indeed was a huge delay, with a train line being closed down for hours. After a slow trickle of information from the train operator, we had to figure out an alternative route on our own, since the DB app was still showing routes that could not be operated for the next few hours because of the train line closure.</p>
<h2 id="conclusion">Conclusion</h2>
<p>There are still two weeks left for the €9 ticket and I’m planning to make the best use of them. So far I’ve been taking a trip on most days, so it’s definitely been worth it and will be missed.</p>
<p>Unfortunately the €9 ticket won’t be prolonged. The high public interest has lead to <a href="https://www.thelocal.de/20220805/how-the-greens-want-to-replace-germanys-e9-ticket-deal/">some plans</a> for a relatively cheap follow-up ticket, but again with some regional limitations for the cheap ticket. <a href="https://www.theguardian.com/money/2022/jul/15/spain-announces-free-rail-journeys-from-september-until-the-end-of-the-year">Other</a> <a href="https://www.vdl.lu/en/getting-around/bus/free-public-transport-and-exceptions">countries</a> are even implementing entirely free public transport meanwhile, so there is some hope for the future.</p>
Update on DoS Attacks against our Online Game2022-05-16T00:00:00+02:00https://hookrace.net/blog/dos-attacks-update
<p>In my <a href="/blog/dos-attacks-against-online-game/">previous post</a> 8 months ago I described how our <a href="https://github.com/ddnet/ddnet">open source</a> online game <a href="https://ddnet.org/">DDraceNetwork</a> has been suffering under DoS attacks for about 8 years, basically since its inception. Recently the attacks have gotten much worse, forcing us to work on further approaches. Since many players made suggestions recently, I’m writing this blog post to summarize what we are attempting and to ask for help again.</p>
<!--more-->
<p><img src="/public/nld.ddnet.tw-net-7d.png" alt="Netherlands" />
<img src="/public/ger2.ddnet.tw-net-49d.png" alt="Germany" /></p>
<p>These traffic graphs are from two of the servers we are running, note the logarithmic x-axis. Each spike represents an incoming DoS attack, as you can see some of them last for nearly a day.</p>
<p>Recently the attacks have been relatively weak in terms of incoming bandwidth, using spoofed IP addresses imitating our UDP-based connection process.</p>
<p>At first the CPU gets overloaded since the server suddenly has to try and handle hundreds of thousands of connection attempts per second. Since our game servers are mostly running on cheap VPSes and each game server runs single-threaded, it is quite easy to overload a system in this manner.</p>
<h2 id="https-based-whitelist">HTTPS-based Whitelist</h2>
<p>To prevent spoofing we collect all players’ IP addresses and whitelist those. Since we also develop the game client, we can modify the client to connect to a server via HTTPs for this whitelisting. The iptables rules and <a href="https://ipset.netfilter.org/">ipset</a> setup on the game servers for this whitelist look something like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ipset create official iphash
ipset create whitelist-ip iphash
iptables <span class="nt">-N</span> serverinfo
iptables <span class="nt">-A</span> serverinfo <span class="nt">-m</span> hashlimit <span class="nt">--hashlimit-above</span> 40/s <span class="nt">--hashlimit-mode</span> dstport <span class="nt">--hashlimit-name</span> si_dstport <span class="nt">-j</span> DROP
iptables <span class="nt">-N</span> game
iptables <span class="nt">-A</span> game <span class="nt">-m</span> <span class="nb">set</span> <span class="nt">--match-set</span> official src <span class="nt">-j</span> ACCEPT
iptables <span class="nt">-A</span> game <span class="nt">-m</span> <span class="nb">set</span> <span class="nt">--match-set</span> whitelist-ip src <span class="nt">-m</span> u32 <span class="nt">--u32</span> <span class="s2">"38=0x67696533"</span> <span class="nt">-j</span> serverinfo
iptables <span class="nt">-A</span> game <span class="nt">-m</span> <span class="nb">set</span> <span class="nt">--match-set</span> whitelist-ip src <span class="nt">-m</span> u32 <span class="nt">--u32</span> <span class="s2">"38=0x66737464"</span> <span class="nt">-j</span> serverinfo
iptables <span class="nt">-A</span> game <span class="nt">-m</span> <span class="nb">set</span> <span class="nt">--match-set</span> whitelist-ip src <span class="nt">-j</span> ACCEPT
<span class="c"># Still allow non-whitelisted players when there is no attack</span>
iptables <span class="nt">-A</span> game <span class="nt">-m</span> limit <span class="nt">--limit</span> 10000 <span class="nt">-j</span> ACCEPT
iptables <span class="nt">-A</span> game <span class="nt">-j</span> DROP
iptables <span class="nt">-A</span> INPUT <span class="nt">-p</span> udp <span class="nt">-m</span> udp <span class="nt">--dport</span> 8000:9000 <span class="nt">-j</span> game</code></pre></figure>
<p>To keep updating the whitelist we run a simple script:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/sh</span>
<span class="nb">rm</span> <span class="nt">-f</span> whitelist whitelist.old
<span class="nb">touch </span>whitelist.old
<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
</span>curl <span class="nt">-o</span> whitelist <span class="nv">$WHITELIST_URL</span>
<span class="nv">LC_ALL</span><span class="o">=</span>C <span class="nb">comm</span> <span class="nt">-23</span> whitelist whitelist.old | <span class="k">while </span><span class="nb">read </span>line<span class="p">;</span> <span class="k">do
</span>ipset add whitelist-ip <span class="nv">$line</span>
<span class="k">done
</span><span class="nb">mv </span>whitelist whitelist.old
<span class="nb">sleep </span>20
<span class="k">done</span></code></pre></figure>
<p>About 20 seconds after starting DDNet client the players are now being whitelisted and can join servers which are under a spoofed DoS attack. We are employing this on the servers which are commonly attacked.</p>
<p>After a few days the ipset runs out of space unfortunately. Since we have ~25k unique players daily, the default maximum element size of 65536 of <code class="language-plaintext highlighter-rouge">ipset</code> is quite tight. To work around this we’ll have to increase the limit or remove older entries automatically.</p>
<p>In order to make this work reliably we had to disable IPv6 for the HTTPS access, since our game servers are IPv4 only at the moment and collecting IPv6 addresses doesn’t help.</p>
<p>An unfortunate realization was that some ISPs don’t provide fixed IPv4 addresses anymore, but instead tunnel their native IPv6 traffic through different IPv4 addresses, based on protocol and port. So we get one IP address in the HTTPS-based whitelist, but another when connecting to the gameserver via UDP. At least one Israeli ISP is doing this.</p>
<h2 id="dedicated-servers">Dedicated Servers</h2>
<p>For some of the servers this whitelisting is not enough, since the hosters’ DoS protection kicks in and starts aggressively blocking UDP traffic. This includes legitimate traffic and results in all players timing out on the game servers. I have tried talking to a few of the hosters about this, but it is by design since they don’t support our game protocol, and wish to protect their own infrastructure as well as other customers. At least they are not kicking us out for now, which has happened in the past with a few hosters after multiple DoS attacks against us impacting their other customers.</p>
<p>So for a few locations we have upgraded to dedicated servers, where the hoster cares less about us exhausting in the incoming network connection and our CPUs for hours at a time. Unfortunately it is difficult to find hosters that provide more than 1 Gbit/s network bandwidth and still fall into our budget of 100 € / month.</p>
<p>The attackers of course notice that these servers can’t be downed as easily, so they switch to different kinds of attacks, exhausting our measly 2 Gbit/s network.</p>
<h2 id="small-hosting-companies">Small Hosting Companies</h2>
<p>Some small hosting companies in Europe offered to help us by providing a free server with custom DoS protection. We accepted two such offers, but didn’t have much success with either of them.</p>
<p>For the first we never managed to get legitimate traffic to get through their firewall.</p>
<p>For the second everything seemed to work fine when there was no attack, but during DoS attacks some player packets would get misflagged all the time. Additionally the network stack had problems and caused our MariaDB-based SQL connection to time out. Unfortunately we didn’t handle this error properly in our code and ended up losing legitimate player ranks.</p>
<p>In the end we didn’t continue using either of these hosters, but spent a few hours trying to set up the firewall with them. I had similar experiences with every paid hoster which offered a custom DoS protection.</p>
<h2 id="dos-protection-companies">DoS Protection Companies</h2>
<p>There are of course also larger companies providing custom DoS protection for UDP-based applications. One of them offered to help us out for free with our problem, and I’ve been in touch with them since the last post. Unfortunately everything is moving very slowly, and many of my requests to move things forward over the weeks have not been answered, so I guess the fact that we are unable to pay thousands of € per month matters after all. So far no custom DoS protection is available here, but I’ll keep pinging them every few weeks.</p>
<h2 id="proxy-servers">Proxy Servers</h2>
<p>Instead of exposing our real game servers directly to the players, for the future we are considering a <a href="https://github.com/ddnet/ddnet/pull/4791">proxy server</a> that sits inbetween the game server and player. This would allow us to run multiple proxies for subsets of the players and attempt to isolate the impact the attacker can have.</p>
<p>This might also allow us to proxy Steam players’ traffic through <a href="https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay">Steam Datagram Relay</a> (SDR), although more work would be required to implement that. Someone from Valve reached out to us after the previous post, and we might follow up on this once the proxy server implementation is ready. It is really cool that Valve is offering help here, even for a free game like ours.</p>
<p>We can’t entirely switch to SDR through Steam’s servers since we want DDNet to stay playable as an open source game. As a solution we can provide a regular open source proxy for open source players, and a separate SDR-based proxy for Steam players.</p>
<h2 id="ipv6-servers">IPv6 Servers</h2>
<p>A longshot idea is using IPv6-only servers. We haven’t tried this yet, maybe the IP spoofing capabilities of our attackers are lower on IPv6. This could also be combined well with the proxy servers by providing an additional ipv6-only proxy server.</p>
<p>Running all these proxy servers for each location will definitely add cost, but it’s probably our best path forward right now. The setup would look something like this for each Server location:
<img src="/public/proxy.png" alt="Graph" /></p>
<figure class="highlight"><pre><code class="language-dot" data-lang="dot"><span class="k">digraph</span> <span class="nv">D</span> <span class="p">{</span>
<span class="k">node</span> <span class="o">[</span><span class="n">shape</span><span class="p">=</span><span class="s2">"box"</span><span class="o">]</span><span class="p">;</span>
<span class="s2">"OS Player 1"</span> <span class="o">-></span> <span class="s2">"IPv4 Proxy"</span> <span class="o">-></span> <span class="s2">"Game Server"</span><span class="p">;</span>
<span class="s2">"OS Player 2"</span> <span class="o">-></span> <span class="s2">"IPv4 Proxy"</span><span class="p">;</span>
<span class="s2">"OS Player 3"</span> <span class="o">-></span> <span class="s2">"IPv6 Proxy"</span> <span class="o">-></span> <span class="s2">"Game Server"</span><span class="p">;</span>
<span class="s2">"OS Player 4"</span> <span class="o">-></span> <span class="s2">"IPv6 Proxy"</span><span class="p">;</span>
<span class="s2">"Steam Player 1"</span> <span class="o">-></span> <span class="s2">"SDR Proxy"</span> <span class="o">-></span> <span class="s2">"Game Server"</span><span class="p">;</span>
<span class="s2">"Steam Player 2"</span> <span class="o">-></span> <span class="s2">"SDR Proxy"</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>If you have any suggestions, or dealt with a similar problem before, we’d be interested to hear from you. You can reach me on <a href="mailto:dennis@felsing.org">dennis@felsing.org</a> as well as on the <a href="https://ddnet.org/discord">DDNet Discord</a> as deen#5910 or on IRC (deen in #ddnet on Quakenet).</p>
macOS Setup after 15 Years of Linux2021-12-26T00:00:00+01:00https://hookrace.net/blog/macos-setup
<p>About 3 years ago I wrote about my <a href="/blog/linux-desktop-setup/">Linux Desktop Setup</a> and having used pretty much the same software setup for 10-15 years. My setup used to be heavily keyboard based. Now for work I am for the first time using a macOS-based Macbook Pro with an M1 Pro CPU. I spent the last 3 weeks using it as a daily driver, tweaking it here and there. Here are my experiences.</p>
<!--more-->
<p><a href="/public/macos/macbookpro.jpg"><img src="/public/macos/macbookpro.jpg" alt="MacBook Pro 16"" /></a>
Hardware-wise the machine is great. With the 10 core ARM64 CPU I get about 2-3 times the CPU performance of my desktop system with an Intel i7 6700k. In <a href="https://ddnet.org/">DDNet</a> I can reach <a href="https://github.com/ddnet/ddnet/pull/4488">1800 FPS</a> which is clearly more than enough and about 10 times the FPS I have with the 6700k’s iGPU. I haven’t heard the fan turn on at all yet, so either my workloads, and even my compilations, are too light-weight, or the fan is pretty quiet. The integrated 120 Hz screen is far better than anything I’ve had before, with clearer colors and smoother motion.</p>
<h2 id="installing-software">Installing Software</h2>
<p>macOS’s AppStore doesn’t seem to have most of the relevant applications for me. <a href="https://brew.sh/">Homebrew</a> seems to be the most popular third-party package manager for macOS, so I chose that, hoping that it would support most programs I want to use. Command line applications can be installed with <code class="language-plaintext highlighter-rouge">brew install</code>, while graphical applications that you want acccessible in the regular Applications directory need a <code class="language-plaintext highlighter-rouge">brew install --cask</code>.</p>
<p>For some applications like <a href="https://mpv.io/">mpv</a> this can be a bit confusing since there is both a <a href="https://formulae.brew.sh/formula/mpv#default">command-line Formulae</a> and a <a href="https://formulae.brew.sh/cask/mpv#default">GUI Cask</a> available for it. The Cask doesn’t support arm64 natively yet, since the <a href="https://laboratory.stolendata.net/~djinn/mpv_osx/">maintainer</a> of it doesn’t have an M1 Mac and mpv doesn’t provide official binaries. So here I noticed a major difference in how Homebrew seems to work compared to Arch Linux for example. On Arch Linux applications are built from source code centrally instead of relying on binaries from a vendor.</p>
<p>Luckily it was easy enough to build a DMG of mpv myself, which I then installed by dragging it into the Applications folder:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://github.com/mpv-player/mpv
<span class="nb">cd </span>mpv
git clean <span class="nt">-f</span> <span class="nt">-d</span>
git pull origin master
./bootstrap.py
<span class="nb">export </span><span class="nv">PKG_CONFIG_PATH</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>brew <span class="nt">--prefix</span> luajit-openresty<span class="si">)</span><span class="s2">/lib/pkgconfig"</span>
./waf configure <span class="nt">--lua</span><span class="o">=</span>luajit
./waf build
TOOLS/osxbundle.py <span class="nt">-s</span> build/mpv</code></pre></figure>
<p>With the <a href="https://github.com/jdek/openwith">openwith</a> utility I can set mpv as the default application for video file types:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openwith io.mpv mkv mov mp4 avi</code></pre></figure>
<p>Compared to the x86-64 binary executed with the Rosetta2 translator this uses half the CPU. Surprisingly playing YouTube videos on Firefox directly is using even less CPU. My current assumption is that the videotoolbox hardware decoding support doesn’t support the same codecs in mpv yet as it does in Firefox.</p>
<p>Update: By building my own FFmpeg version <a href="https://github.com/FFmpeg/FFmpeg/commit/a41a2efc85f8c88caec10040ee437562f9d0b947">VP9 videotoolbox support</a> is available, it will probably be in the next FFmpeg release.</p>
<h2 id="command-line-software">Command line software</h2>
<p>My regular command line software like vim, mutt, remind, htop, unison, yt-dlp, newsboat, ssh, rsync, <a href="https://github.com/def-/rrb">rrb</a> work just fine and were easy to install with a <code class="language-plaintext highlighter-rouge">brew install</code> command. So most of my setup can stay the same and I can still have the same setup whether I run my applications in a local terminal or on my home server via SSH, for example from my phone or a remote machine.</p>
<h2 id="window-management">Window management</h2>
<p>I started missing <a href="https://xmonad.org/">xmonad</a>, my tiling window manager of choice immediately. So my first idea was to use <a href="https://ianyh.com/amethyst/">Amethyst</a> instead, which provides some of the features of xmonad, but without the customizability of writing your own config in <a href="https://www.haskell.org/">Haskell</a>.</p>
<p>It seemed to work ok, but was feeling a bit more awkward to use compared to on Linux. So I decided to use the native macOS window management instead and try to get used to it instead of actively fighting against it. At the moment I’m using the integrated 16” screen and an external 24” monitor. Most applications I use “zoomed in”, which means they take all the space, but are not actually fullscreen, so you still see the dock and window’s title bar. There appears to be no system shortcut to zoom a window, so I have to double click the title bar.
<a href="/public/macos/screen.png"><img src="/public/macos/screen.png" alt="screen" /></a></p>
<p>Update: I found out that you can set a shortcut for “Zoom” by adding it in the System Preferences. This apparently works for any term that is available in the program’s title bar:
<a href="/public/macos/zoom.png"><img src="/public/macos/zoom.png" alt="zoom" /></a></p>
<p>Update 2: Several people suggested <a href="https://rectangleapp.com/">Rectangle</a> for moving and resizing windows, which seems to work well, especially with my large 40” screen.</p>
<h2 id="terminal">Terminal</h2>
<p><a href="https://iterm2.com/">iTerm2</a> is the terminal emulator I chose to use. It has lots of features and so far seems to work fine. Each window and tab gets a dedicated shortcut, and it allows splitting the tab horizontally and vertically, so I can have multiple terminals open at once and jump between them reasonably without needing to use the mouse.</p>
<p>I mostly have one iTerm window open for most command line applications. If I need something quickly, I bound command-return to open a hotkey terminal window. Opening a new iTerm window from outside the application was slightly more challenging. The easiest way is pressing command-n from inside iTerm. From outside just using F4 and writing iTerm will do nothing when another iTerm window is already open. I had to set up an AppleScript workflow in Automator:
<a href="/public/macos/automator.png"><img src="/public/macos/automator.png" alt="automator" /></a></p>
<figure class="highlight"><pre><code class="language-applescript" data-lang="applescript"><span class="k">on</span> <span class="nb">run</span><span class="w"> </span><span class="p">{</span><span class="nv">input</span><span class="p">,</span><span class="w"> </span><span class="nv">parameters</span><span class="p">}</span><span class="w">
</span><span class="k">tell</span><span class="w"> </span><span class="nb">application</span><span class="w"> </span><span class="s2">"iTerm"</span><span class="w">
</span><span class="nv">create</span><span class="w"> </span><span class="na">window</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">default</span><span class="w"> </span><span class="nv">profile</span><span class="w">
</span><span class="k">end</span><span class="w"> </span><span class="k">tell</span><span class="w">
</span><span class="nb">return</span><span class="w"> </span><span class="nv">input</span><span class="w">
</span><span class="k">end</span><span class="w"> </span><span class="nb">run</span></code></pre></figure>
<p>Then it can be set as a global shortcut in Preferences:
<a href="/public/macos/shortcut.png"><img src="/public/macos/shortcut.png" alt="shortcut" /></a></p>
<p>There seems to be considerable overhead to using Automator for shortcuts though, the terminal window takes nearly a second to appear. Luckily I rarely needed a new terminal, since the windows persist after a reboot and I mostly just have to set them up once, and the hotkey terminal is fine for quick tasks like checking mail.</p>
<p>One issue I still have is that iTerm seems to send the home/end/insert keys differently from urxvt, my terminal emulator on Linux. But since both are identifying as <code class="language-plaintext highlighter-rouge">TERM=xterm-256color</code> to support full colors, I can’t tell them apart using zkbd when SSHing into a remote system. Not sure yet how to solve this cleanly. Here are the differences:
<a href="/public/macos/zkbd.png"><img src="/public/macos/zkbd.png" alt="zkbd" /></a></p>
<p>My favorite fixed-width bitmap font <a href="http://terminus-font.sourceforge.net/">Terminus</a> works fine with anti-aliasing disabled in iTerm with the <a href="https://github.com/Homebrew/homebrew-cask-fonts/blob/master/Casks/font-terminus.rb">font-terminus formula</a> installed via:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">brew tap homebrew/cask-fonts
brew <span class="nb">install </span>font-terminus</code></pre></figure>
<p><a href="/public/macos/terminus.png"><img src="/public/macos/terminus.png" alt="terminus" /></a></p>
<p>Unfortunately the font only works at specific sizes, I haven’t figured out yet how to exclude them when changing the font-sizes on-the-fly using command-+ and command–. But I rarely need to change font sizes anyway.</p>
<p>I still use my <a href="https://github.com/def-/tach">tach script</a> so I don’t use my terminals when I accidentally close the iTerm windows:</p>
<blockquote>
<p>tach is a script that provides automatic detaching and reattaching of terminals using dtach</p>
</blockquote>
<p>This also allows me to move a running terminal session from one place to another using only the keyboard by detaching it using command-w and opening a new terminal anywhere I want, which will contain the previously detached session.</p>
<h2 id="keyboard-layout">Keyboard layout</h2>
<p>I’ve been using a German keyboard layout for a while, since a short <a href="https://sourceforge.net/projects/dva/">excursion to Dvorak</a>. My first problem was that macOS’s German keyboard layout uses option-l instead of the option-q I was used to for typing the @ character. Luckily this is easily remedied by switching from the “German” keyboard layout to “German - Standard”.</p>
<p><a href="/public/macos/layout.png"><img src="/public/macos/layout.png" alt="keyboard layout selection" /></a></p>
<p>Unfortunately this is still not quite the same as on a German standard PC keyboard, where the altgr key sits just right of the spacebar, while on a Mac the option key is seperated from the spacebar by the command key. The problem is exacerbated by command-q being the default keyboard shortcut to close an application on macOS. So I ended up closing my currently open application every time I tried entering an @, for example when writing my own email address trying to log into a website.</p>
<p>That’s not ideal, so I thought the easiest solution would be switching the command and option modifier keys in the System Preferences:</p>
<p><a href="/public/macos/modifier.png"><img src="/public/macos/modifier.png" alt="modifier key switch" /></a></p>
<p>This brought me one step forward, two steps back: Now the default system key binds are more akward, since this also switches the left option and command keys. Apple uses command-f (search), command-c (copy), command-v (paste) etc. and all of these became a bit more awkward to press with the preferences-based modifier key switch.</p>
<p>Additionally I noticed that the left option key was actually switched with the right command key and the left command key with the right option key, which is also not ideal when you have programs that only listen to one of the modifiers. At least I got my first macOS bug report out of this.</p>
<p>A better solution for me was using <a href="https://karabiner-elements.pqrs.org/">Karabiner-Elements</a> to only swap the right command and option keys:
<a href="/public/macos/karabiner.png"><img src="/public/macos/karabiner.png" alt="karabiner config" /></a></p>
<p>Of course I also remapped capslock as a secondary escape key, which is commonly required in Vim to leave insert mode. This also worked fine in the System Preferences for what it’s worth.</p>
<p>It’s unfortunate this modifier remapping has to be done with an external application, which can now read all my keyboard inputs system-wide, and thus has to be trusted.</p>
<p>Update: Thanks to Anriudh for suggesting an easier <a href="https://hidutil-generator.netlify.app/">solution</a> for the remapping. This file in <code class="language-plaintext highlighter-rouge">~/Library/LaunchAgents/com.local.KeyRemapping.plist</code> takes care of the problem, no more Karabiner-Elements required:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="cp"><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"></span>
<span class="nt"><plist</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>Label<span class="nt"></key></span>
<span class="nt"><string></span>com.local.KeyRemapping<span class="nt"></string></span>
<span class="nt"><key></span>ProgramArguments<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/usr/bin/hidutil<span class="nt"></string></span>
<span class="nt"><string></span>property<span class="nt"></string></span>
<span class="nt"><string></span>--set<span class="nt"></string></span>
<span class="nt"><string></span>{"UserKeyMapping":[
{
"HIDKeyboardModifierMappingSrc": 0x7000000E7,
"HIDKeyboardModifierMappingDst": 0x7000000E6
},
{
"HIDKeyboardModifierMappingSrc": 0x7000000E6,
"HIDKeyboardModifierMappingDst": 0x7000000E7
},
{
"HIDKeyboardModifierMappingSrc": 0x700000039,
"HIDKeyboardModifierMappingDst": 0x700000029
}
]}<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"><key></span>RunAtLoad<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"></dict></span>
<span class="nt"></plist></span></code></pre></figure>
<h2 id="deadkeys">Deadkeys</h2>
<p>On Linux I’ve always been using a nodeadkeys keyboard layout, which means keys like ^ and ´ get output immediately instead of being combined with the next letter entered. There seems to be no support for this on macOS, so I had to make my own keyboard layout using <a href="https://software.sil.org/ukelele/">Ukelele</a>:
<a href="/public/macos/ukelele.png"><img src="/public/macos/ukelele.png" alt="ukelele" /></a></p>
<p>For remapping modifier keys Ukelele recommends still using Karabiner, so unfortunately we can’t solve all keyboard layout problems with a single new keyboard layout:</p>
<blockquote>
<p>Warning!</p>
<p>Ukelele can change the output of special keys, but this doesn’t always work as expected, or for all applications. There are usually other ways to achieve what you want.</p>
<p>• If you want to rearrange the modifier keys, try System Preferences.</p>
<p>• If you want to make other modifications such as turning modifier keys into normal keys, or normal keys into modifier keys, Karabiner (https://pqrs.org/osx/karabiner/) can likely help you.</p>
<p>• If you would like to be able to use the F-keys (F1 through F19), then Better Touch Tool (https://www.boastr.net/) can probably help you.</p>
<p>This warning appears once per session, and does not stop you changing special key output.</p>
</blockquote>
<table>
<tbody>
<tr>
<td>Ukelele also allowed me to change command-spacebar from outputting a non breaking space (aka <code class="language-plaintext highlighter-rouge">&nbsp;</code> in HTML) to a regular space. It was easy to still have the option key pressed after writing a</td>
<td>symbol on the German keyboard layout, so the following space often became a non breaking space for me, which threw off code linters.</td>
</tr>
</tbody>
</table>
<p>Finally the created <a href="/public/macos/German%20Standard%20NDK.bundle.zip">keyboard layout bundle</a> file can be copied into /Library/Keyboard Layouts/ and then be selected in the system settings. It seems not to work during the Login screen, but I don’t mind much.</p>
<h2 id="mouse">Mouse</h2>
<p>I’m using a Logitech MX Master 3 connected via Bluetooth. I want it to scroll normally (up means up, down means down). But I want to keep the “natural” scrolling direction of the touch pad, which works like scrolling on a smart phone, so you move two fingers up to scroll down, two fingers down to scroll up. With the system-wide settings it seems only possible to change this for both the touchpad and the mouse at once.</p>
<p><a href="https://www.logitech.com/en-us/product/options">Logitech Options</a> luckily supports changing this just for the mouse, but always takes a few seconds at startup to register the device and start working.
<a href="/public/macos/logioptions.png"><img src="/public/macos/logioptions.png" alt="logioptions" /></a></p>
<p>Unfortunately the mouse scrolling seems to have an acceleration, which I have not been able to disable yet.</p>
<p>Update: Luc sent me this command which seems to improve the acceleration situation, but it still doesn’t feel as linear as on Linux:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">defaults write .GlobalPreferences com.apple.scrollwheel.scaling <span class="nt">-1</span></code></pre></figure>
<p>Update: Thanks to Kristjan for suggesting <a href="https://linearmouse.org/">LinearMouse</a>, which solves the acceleration problem for me. No more Logitech Options required either to reverse scrolling wheel on mouse only.</p>
<p>Update 2: LinearMouse is somehow limiting my mouse wheel speed when scrolling very quickly, so I still prefer Logitech Options.</p>
<p>Putting the display to sleep easily by moving the mouse cursor in the corner is nice (I’d still prefer a keyboard shortcut), and can be enabled in the system preferences:
<a href="/public/macos/hotcorner.png"><img src="/public/macos/hotcorner.png" alt="hotcorner" /></a></p>
<h2 id="task-bar">Task bar</h2>
<p>I use <a href="https://github.com/exelban/stats">Stats</a> to display some network and CPU statistics in the task bar, works solidly so far and has replaced <a href="https://github.com/brndnmtthws/conky">conky</a> for me.
<a href="/public/macos/stats.png"><img src="/public/macos/stats.png" alt="stats" /></a></p>
<h2 id="firefox--web">Firefox & Web</h2>
<p>Safari doesn’t support <a href="https://github.com/gorhill/uBlock">uBlock Origin</a>, see this <a href="https://github.com/el1t/uBlock-Safari/issues/158">explanation by ghost</a>:</p>
<blockquote>
<p>Very quick tl;dr: uBO will no longer work with Safari, use Firefox or a new “content blocker” app (see below for good replacements).</p>
</blockquote>
<p>So this made the choice easy for me, since Chrome will also <a href="https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2#c23">phase out uBlock Origin support</a>. I simply kept using Firefox with my essential addons: uBlock Origin and my password manager <a href="https://bitwarden.com/">Bitwarden</a>.</p>
<p>I figured out that it’s quite easy to have my command line email client <a href="http://www.mutt.org/">mutt</a> preview HTML emails inline, but still open Firefox by pressing m on the HTML part, by adding this to my .mailcap file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>text/html; /Applications/Firefox.app/Contents/MacOS/firefox '%s' &; needsterminal
text/html; links -dump %s; needsterminal; copiousoutput;
</code></pre></div></div>
<p>To handle URLs and other texts copied on the command line automatically, pbpaste/pbcopy replace xclip as the clipboard access tools on macOS. Downloading a video from the URL in clipboard for example:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">pbpaste|xargs yt-dlp <span class="nt">--</span></code></pre></figure>
<h2 id="music">Music</h2>
<p>I tried using the integrated Music application, but it kept struggling to import my music database. This might have been slow since I connected to it using <a href="https://osxfuse.github.io/">macFUSE & SSHFS</a>. I don’t need a fancy music player anyway and remembered that > 15 years ago I used to use <a href="https://www.foobar2000.org/">Foobar2000</a> on Windows. It’s still around and there is a macOS version, so I’m just using that for now.
<a href="/public/macos/foobar2000.png"><img src="/public/macos/foobar2000.png" alt="foobar2000" /></a></p>
<p>With a longer read-ahead I haven’t had any pauses in music playback so far, even though the sshfs connection is over the internet. Sound quality of the MacBook is fine, I haven’t noticed any problems and the dedicated headphone jack is appreciated (as is the half-size SD card slot for loading music onto the SD card for my car’s stereo).</p>
<p>The audio volume controls are a bit too coarse-grained for me, but using shift-option-F11/F12 allows for a few more steps when setting the volume.</p>
<h2 id="translations">Translations</h2>
<p>I used to use <a href="https://git.sr.ht/~tsdh/rdictcc">rdictcc</a> to easily access the excellent German-English translations from wiki-like dictionary service <a href="https://www.dict.cc/">dict.cc</a>. There is a <a href="https://www.dict.cc/?s=about%3Awordlist&l=e">dict.cc plugin</a> for the macOS dictionary which I am now using instead.</p>
<p>On macOS translations are easily accessible by holding the mouse over a word or a selected term and pressing control-command-d.
<a href="/public/macos/translations.png"><img src="/public/macos/translations.png" alt="translations" /></a></p>
<h2 id="standby">Standby</h2>
<p>To get system standby not to wake up constantly I had to disable “Wake for network access” in Battery settings:
<a href="/public/macos/standby.png"><img src="/public/macos/standby.png" alt="standby" /></a>
I also previously set</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">defaults write /Library/Preferences/com.apple.mDNSResponder.plist NoMulticastAdvertisements <span class="nt">-bool</span> YES</code></pre></figure>
<p>since the DNS responder kept waking up the system according to <code class="language-plaintext highlighter-rouge">pmset -g log|grep "Wake Requests"</code>, but this might also be fixed by the “Wake for network access” setting.</p>
<h2 id="useful-shortcuts">Useful shortcuts</h2>
<p>Here are some of the default system shortcuts I’ve been using to get over not using xmonad anymore as my window manager (and Vimperator/Pentadactyl for Firefox respectively, but that switch I already had to make a few years ago):</p>
<p><a href="/public/macos/keyboard.jpg"><img src="/public/macos/keyboard.jpg" alt="keyboard" /></a></p>
<table>
<tbody>
<tr>
<td>Command</td>
<td>Application</td>
<td>Shortcut</td>
</tr>
<tr>
<td>Open app</td>
<td> </td>
<td>🌐 F4</td>
</tr>
<tr>
<td>Next app</td>
<td> </td>
<td>⌘ ⇥</td>
</tr>
<tr>
<td>Previous app</td>
<td> </td>
<td>⇧ ⌘ ⇥</td>
</tr>
<tr>
<td>Next desktop</td>
<td> </td>
<td>⌃ ►</td>
</tr>
<tr>
<td>Previous desktop</td>
<td> </td>
<td>⌃ ◄</td>
</tr>
<tr>
<td>Mission control</td>
<td> </td>
<td>⌃ ▲ / 🌐 F3</td>
</tr>
<tr>
<td>Move app to next desktop</td>
<td> </td>
<td>Click and hold title bar, ⌃ ►</td>
</tr>
<tr>
<td>Move app to previous desktop</td>
<td> </td>
<td>Click and hold title bar, ⌃ ◄</td>
</tr>
<tr>
<td>New window</td>
<td> </td>
<td>⌘ N</td>
</tr>
<tr>
<td>Close window</td>
<td> </td>
<td>⌘ Q</td>
</tr>
<tr>
<td>Move focus to next window</td>
<td> </td>
<td>⌘ `</td>
</tr>
<tr>
<td>New tab</td>
<td>iTerm, Firefox</td>
<td>⌘ T</td>
</tr>
<tr>
<td>Close tab</td>
<td>iTerm, Firefox</td>
<td>⌘ W</td>
</tr>
<tr>
<td>Next tab</td>
<td>iTerm, Firefox</td>
<td>⌃ ⇥</td>
</tr>
<tr>
<td>Previous tab</td>
<td>iTerm, Firefox</td>
<td>⇧ ⌃ ⇥</td>
</tr>
<tr>
<td>Restore tab</td>
<td>Firefox</td>
<td>⇧ ⌘ T</td>
</tr>
<tr>
<td>Split vertically</td>
<td>iTerm</td>
<td>⌘ D</td>
</tr>
<tr>
<td>Split horizontally</td>
<td>iTerm</td>
<td>⇧ ⌘ D</td>
</tr>
<tr>
<td>Select upper pane</td>
<td>iTerm</td>
<td>⌥ ⌘ ▲</td>
</tr>
<tr>
<td>Select lower pane</td>
<td>iTerm</td>
<td>⌥ ⌘ ▼</td>
</tr>
<tr>
<td>Select right pane</td>
<td>iTerm</td>
<td>⌥ ⌘ ►</td>
</tr>
<tr>
<td>Select left pane</td>
<td>iTerm</td>
<td>⌥ ⌘ ◄</td>
</tr>
<tr>
<td>Increase font size</td>
<td>iTerm, Firefox</td>
<td>⌘ +</td>
</tr>
<tr>
<td>Decrease font size</td>
<td>iTerm, Firefox</td>
<td>⌘ -</td>
</tr>
<tr>
<td>Search</td>
<td> </td>
<td>⌘ F</td>
</tr>
<tr>
<td>Copy</td>
<td> </td>
<td>⌘ C</td>
</tr>
<tr>
<td>Cut</td>
<td> </td>
<td>⌘ X</td>
</tr>
<tr>
<td>Paste</td>
<td> </td>
<td>⌘ V</td>
</tr>
<tr>
<td>Save</td>
<td> </td>
<td>⌘ S</td>
</tr>
<tr>
<td>Select all</td>
<td> </td>
<td>⌘ A</td>
</tr>
<tr>
<td>Select URL field</td>
<td>Firefox</td>
<td>⌘ L</td>
</tr>
<tr>
<td>Open URL</td>
<td>iTerm</td>
<td>⌘ Left mouse click</td>
</tr>
<tr>
<td>Dictionary lookup</td>
<td> </td>
<td>⌃ ⌘ D</td>
</tr>
<tr>
<td>Take screenshot of area</td>
<td> </td>
<td>⇧ ⌘ 4</td>
</tr>
<tr>
<td>Take screenshot of window</td>
<td> </td>
<td>⇧ ⌘ 4, ⎵</td>
</tr>
<tr>
<td>Increase volume slightly</td>
<td> </td>
<td>⇧ ⌥ F12</td>
</tr>
<tr>
<td>Decrease volume slightly</td>
<td> </td>
<td>⇧ ⌥ F11</td>
</tr>
</tbody>
</table>
<p>And since some keys are missing on the MacBook keyboard here is how to get them:</p>
<table>
<tbody>
<tr>
<td>Key</td>
<td>Shortcut</td>
</tr>
<tr>
<td>Page up</td>
<td>🌐 ▲</td>
</tr>
<tr>
<td>Page down</td>
<td>🌐 ▼</td>
</tr>
<tr>
<td>Home</td>
<td>🌐 ◄</td>
</tr>
<tr>
<td>End</td>
<td>🌐 ►</td>
</tr>
<tr>
<td>Delete</td>
<td>🌐 ⌫</td>
</tr>
<tr>
<td>Insert</td>
<td>🌐 ⏎</td>
</tr>
</tbody>
</table>
<h2 id="conclusion">Conclusion</h2>
<p>macOS is working better than expected for me so far, and definitely better than Windows would. So I’m glad with my choice of a MacBook Pro as a work device. Now that I’ve written down some of the steps I had to take to make it more usable, I hopefully won’t forget them if I ever have to set a macOS based system up again. If you have any questions or tips get in touch with me at <a href="mailto:dennis@felsing.org">dennis@felsing.org</a>.</p>
<p>Discussion on <a href="https://news.ycombinator.com/item?id=29742551">Hacker News</a>.</p>
DoS Attacks against our Online Game2021-09-27T00:00:00+02:00https://hookrace.net/blog/dos-attacks-against-online-game
<p><a href="https://ddnet.org/">DDraceNetwork</a> is an <a href="https://github.com/ddnet/ddnet">open source</a> online game I’ve been running since 2013 with a community of volunteers. The game is available for free, I’m hosting servers for it in many countries <a href="https://ddnet.org/status/">around the world</a> so that we have trusted <a href="https://ddnet.org/ranks/">official ranks</a>. The servers are paid for by <a href="https://ddnet.org/funding/">donations</a>, which I stop collecting once the cost of the servers for the current year is covered. I wrote about DDNet in a <a href="/blog/ddnet-evolution-architecture-technology/">previous post in 2016</a> and also a bit about the <a href="https://forum.ddnet.org/viewtopic.php?t=1824">game history in 2013</a>.</p>
<p>Pretty much since the beginning we have been suffering from DoS attacks against the servers. Since the <a href="https://store.steampowered.com/app/412220/DDraceNetwork/">Steam release</a> in 2020 the player number has increased significantly, so that we have <a href="https://ddnet.org/stats/">about 1300 players</a> playing on average.</p>
<!--more-->
<p><a href="https://ddnet.org/stats/"><img alt="Points earned per Day by Server Type" src="/public/points-earned.svg" style="width: 100%;" /></a></p>
<p>Based on the rising player number the urge to deal with the DoS problem is larger than ever. A few months ago for example we organized a <a href="https://ddnet.org/tournaments/56/">Tournament</a> for everyone in the community to participate. Unfortunately the event was the continuous target of DoS attacks for multiple hours, with players fleeing from one server to the next, trying to find a safe refuge. These <a href="https://ddnet.org/stats/server/">bandwidth graphs</a> for our servers in Netherlands and Germany respectively illustrate the problem:</p>
<p><img src="/public/nld.ddnet.tw-net-1d.png" alt="Netherlands" />
<img src="/public/ger2.ddnet.tw-net-1d.png" alt="Germany" /></p>
<p>Since DDNet only runs online and you feel every small network latency it is predestined as a target for DoS attacks. Outside of attacks our servers don’t need to be particularly powerful for regular gameplay, 1 CPU core can support 150 players concurrently, memory consumption is minimal. So the total we are paying for our ~23 servers around the world is only <a href="https://ddnet.org/funding/">~2300 € for the year 2021</a>.</p>
<h2 id="what-we-have-tried">What we have tried</h2>
<p>The kinds of attacks we receive are varied. There are generic reflection and amplification attacks, as well as spoofed game specific attacks of considerable strength. Some hosters have thrown us out following attacks impacting their other customers, and most hosters will blackhole our IP address for multiple hours before it gets to that.</p>
<p>There is even a <a href="https://github.com/heinrich5991/libtw2/tree/master/wireshark-dissector">Wireshark dissector</a> for DDNet/Teeworlds available, written in Rust:</p>
<p><img src="/public/dissector.png" alt="Wireshark Dissector Screenshot" /></p>
<p>Some components of our system were relatively easy to protect: The database server has been moved to a secret IP address, and the game servers will store ranks in a local SQLite file if the main MySQL-based database is not available.</p>
<p>The web server hosting <a href="https://ddnet.org/">DDNet.tw</a> and other HTTPS-based components like the map downloader and updater are now reachable through Cloudflare only, and have received no attacks since then.</p>
<p>For the individual server infos the client currently has to communicate with each game server by UDP, thus revealing its own IP address without having connected to a server. Since one of the known attackers is running their own DDNet server, they can use this method to collect legitimate player IP addresses and spoof them in their attacks. To prevent this heinrich5991 implemented a centrally provided <a href="https://github.com/ddnet/ddnet/pull/3772/">HTTPS-based server info</a>. This means that every player only has to access the <a href="https://master1.ddnet.org/ddnet/15/servers.json">JSON file</a> containing all server information, instead of previously receiving this information via UDP from each server directly.</p>
<p>To prevent our own servers being used for amplification attacks we limit the server info packets per IP address in our <a href="https://github.com/ddnet/ddnet-scripts/blob/892c22672d93c1ea78e24ef81fa09ff60d28fdcc/ddnet-setup.sh#L29-L38">server setup script</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">iptables <span class="nt">-t</span> raw <span class="nt">-A</span> PREROUTING <span class="nt">-p</span> udp <span class="nt">-j</span> NOTRACK
iptables <span class="nt">-t</span> raw <span class="nt">-A</span> OUTPUT <span class="nt">-p</span> udp <span class="nt">-j</span> NOTRACK
iptables <span class="nt">-N</span> serverinfo
iptables <span class="nt">-A</span> INPUT <span class="nt">-p</span> udp <span class="nt">-m</span> u32 <span class="nt">--u32</span> <span class="s2">"38=0x67696533"</span> <span class="nt">-j</span> serverinfo
iptables <span class="nt">-A</span> INPUT <span class="nt">-p</span> udp <span class="nt">-m</span> u32 <span class="nt">--u32</span> <span class="s2">"38=0x66737464"</span> <span class="nt">-j</span> serverinfo
iptables <span class="nt">-A</span> serverinfo <span class="nt">-m</span> hashlimit <span class="nt">--hashlimit-above</span> 1000/s <span class="nt">--hashlimit-burst</span> 2500 <span class="nt">--hashlimit-mode</span> dstport <span class="nt">--hashlimit-name</span> si_dstport <span class="nt">-j</span> DROP
iptables <span class="nt">-A</span> serverinfo <span class="nt">-m</span> hashlimit <span class="nt">--hashlimit-above</span> 20/s <span class="nt">--hashlimit-burst</span> 100 <span class="nt">--hashlimit-mode</span> srcip <span class="nt">--hashlimit-name</span> si_srcip <span class="nt">-j</span> DROP
iptables-save <span class="o">></span> /etc/iptables.up.rules</code></pre></figure>
<p>Unfortunately even sending a few packets per second is considered a lot by some people, so with the spoofed server info requests coming in, we get a lot of complaints for the “port scanning” that we seem to be doing. I have to react quickly to those abuse mails in order to prevent our servers from being shut down, having to explain the situation and blocking any traffic from the IP range in question. An example mail:</p>
<blockquote>
<p>On 2021-04-19T14:58+0200, abuse@hetzner.com wrote:</p>
<blockquote>
<p>On 18 Apr 12:03, abuse-out@X.com wrote:</p>
<blockquote>
<p>The address 49.12.97.180 from Your network tried to log in to
our network using Port […]</p>
<p>Below You will find a listing of the dates and
times the incidents occured as well as the attacked IP-Addresses.
This is a matter of concern for us and continued tries might result in
legal action. If the machine was victim to a hack take it offline, repair
the damage and use better protection next time.
The times included are in Central European Time (CET).</p>
<p>[…]</p>
<p>Regards,</p>
<p>X</p>
</blockquote>
</blockquote>
<blockquote>
<p>Dear Mr Dennis Felsing,</p>
<p>We have received information regarding spam and/or abuse from abuse-out@X.com
Please take all necessary measures to avoid this in the future.</p>
<p>We also request that you send a statement within 24 hours to us and to the person who filed the complaint. This
response should contain information about how this could have happened and what you intend to do about it.</p>
<p>How to proceed:</p>
<ul>
<li>Solve the issue</li>
<li>Send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=X</li>
<li>Send a response by email to the person who filed the complaint</li>
</ul>
<p>The statement will be checked by a staff member who will then coordinate any further proceedings. If you fail to
comply within the stated deadline, the IP may be blocked.</p>
<p>Important note:
When replying to us, please leave the abuse ID [AbuseID:X] unchanged in the subject line.</p>
<p>Kind regards</p>
<p>X</p>
</blockquote>
<p>Hi Mr. X,</p>
<p>The server in question (49.12.97.180) only responds to incoming requests. We received an incoming DoS attack with
spoofed IP addresses, so the server replied to those. We already limit the number of packets per server that we respond
to. For the future I have blocked all packets from X/24 of reaching the server, so the server also won’t send
any packets to them.</p>
<p>Best regards</p>
<p>Dennis Felsing</p>
</blockquote>
<p>When players get a timeout ingame, they can reconnect later and keep playing from where they were before. We tried detecting a DoS attack and prolonging the timeout protection to <a href="https://github.com/ddnet/ddnet/issues/3780">1 hour</a>, but this lead to the problem of servers being full of characters having a timeout, making it impossible for new players to join, even those trying to reclaim their timed out character. Additionally teams can use the save functionality to save their progress and continue it at any later time, or even on another server.</p>
<p>This doesn’t help when a VPS gets nullrouted for hours or even days at a time due to attacks. Many hosters offer “DDoS protection” for their servers, but so far none worked well. Some block all UDP traffic during an attack, which is clearly suboptimal for a UDP-based game. Others try to train their DDoS detection a bit to detect legitimate traffic, but some players and some packets of all players will still be considered as part of the attack and get blocked. The attackers can even use the DDoS protection and spoof the collected IP addresses of real players to get specific player IP addresses blocked. As a reaction of that I then have to discuss with the hoster for a long time that the players probably are not part of a botnet, but being spoofed.</p>
<p>Instead of cheap VPS servers we have tried getting dedicated servers at larger European hosters like OVH, Hetzner, ihor and NFOrce. The idea is that we have exclusive resources, so the chances of us impacting other customers is lower, and thus we won’t get nullrouted so easily. Largely this works, but the available network bandwidth (usually 1-10 Gbit/s) as well as CPU usage become the limit. Each individual game server uses a single thread to handle all network packets, so it is still relatively easy and cheap for attackers to overload.</p>
<p>Cloud-based servers at AWS or Google Cloud perform well, even during attacks, but the bandwidth cost is exorbitant. We could consider using them for short periods during tournaments instead of permanently though.</p>
<h2 id="what-we-can-still-try">What we can still try</h2>
<p>So far we have only tried 10 Gbit/s dedicated servers. Maybe the reason that the Cloud providers function so well during DoS attacks is that the actual servers the virtual machines run on have even larger bandwidth available. So we could still try to go to 20-25 Gbit/s for some dedicated servers, but only few cheap hosters offer this and what they charge for it is a lot for us. An example would be an additional 50 € booking fee and 100 € more per month for 20 Gbit/s at <a href="https://www.nforce.com/">NFOrce</a></p>
<p>The network handling of the game servers can be switched to run multithreaded, so that we can at least handle the packets of unconnected players faster. This doesn’t help with the really cheap servers, and the attacker could just increase the attack strength beyond what we can handle with our usual 2-4 available cores. This already happens when multiple game servers are attacked at once, so it seems like it might not help much.</p>
<p>Instead of improving the game servers, we could have a Linux kernel based firewall using Express Data Path (XDP) for higher performance. Other server hosters, namely noby and ReiTW, had success with this approach, but for us this hasn’t helped much yet, probably since we are still out of bandwidth and CPU.</p>
<p>So the next logical step would be not to have this kind of firewall on our rented virtual machines, but as part of the dedicated DoS solutions that the hosters are providing. Unfortunately so far no hoster was willing to add it, except for a large fee. Hosting a more popular online game would make this easier, maybe we should just imitate another game’s network packets to benefit from the improved DoS protection support.</p>
<p>Instead of another UDP-based game’s network packets we also have a way to communicate using Websockets instead. Potentially Cloudflare or other web-oriented DoS protections could protect that well. Unfortunately Websockets are TCP-based and thus not perfectly suited for a fast-paced game where it makes no sense to resend out-of-date state updates from server to client.</p>
<p>Of course there are proper solutions that offer DoS protection for UDP-based games, such as Cloudflare Gaming or Akamai Gaming, but their costs start at thousands of € per month. Thus we clearly can’t afford them.</p>
<p>Instead of technical solutions, we could try working with the police. We know the real name of the main DoS attacker, but there is no way to link him to the attacks from the data we have. Even then, this might not help much. I have previously reported a DoS attacker to the police, since he was kind enough to send a message claiming an attack as his own from his real phone number. But since the attacker was a minor and we are a free online game and thus have no measurable economic damage, the public prosecutor left it at a sternly worded warning.</p>
<h2 id="the-unknown">The Unknown</h2>
<p>I wrote most of this post back in May when we were facing large-scale attacks the last time. Since then not much happened, so I didn’t want to bring any unwanted attention to our situation. Now in September the attacks seem to be starting again with a long downtime on all our servers today, so I decided to finish this up and post it anyway. Let’s hope that this brings more positive attention than negative. After all, I just showed exactly how good of a DoS target we are.</p>
<p>If you have any suggestions, or dealt with a similar problem before, we’d be interested to hear from you. You can reach me on <a href="mailto:dennis@felsing.org">dennis@felsing.org</a> as well as on the <a href="https://ddnet.org/discord">DDNet Discord</a> as deen#5910 or on IRC (deen in #ddnet on Quakenet). I hope that there is something trivial we are missing.</p>
<p>Discussion on <a href="https://news.ycombinator.com/item?id=28675094">Hacker News</a>.</p>
Quest for the Lost Home Server2019-10-15T00:00:00+02:00https://hookrace.net/blog/quest-for-the-lost-home-server
<p>Today I lost access to my home server. As I described in <a href="/blog/linux-desktop-setup/">a previous post</a> I depend heavily on the server to fetch my emails, as a file server, to synchronize files, for newsbeuter and irssi sessions and many other things. As no one was going to be in proximity of the server for the next few hours, my goal for today was to solve the problem remotely.</p>
<!--more-->
<p>The symptom was that my SSH connection attempts failed. The server also didn’t respond to pings. As the server is sitting at home behind a regular DSL connection it uses a dynamic IP address that is shuffled every 24 hours. So my first hunch was that the daily reconnect might just have happened at a different time today and I gave the server some time to broadcast its new IP address to my domain registrar.</p>
<p>After about an hour I still couldn’t connect, so I started investigating. Maybe the API of the domain registrar changed (as happened before) or my script failed for another reason? I know that my home server does nightly backups of the other servers I run. So I connected to one of them and checked the journalctl log. To my surprise no connection from the server happened last night. My worst fear was that the server was hanging due to a hardware problem, as I ran into <a href="https://bugzilla.kernel.org/show_bug.cgi?id=109051">similar problems</a> with this Intel Bay Trail CPU before. (The issue seems to be that Intel underdesigned the power delivery on those systems, which the graphics driver is trying to <a href="https://cgit.freedesktop.org/drm-intel/commit/?id=a75d035fedbdecf83f86767aa2e4d05c8c4ffd95">work around</a>.)</p>
<p>But I wasn’t ready to give up yet, so I tried to think of any other activities the server would do that might leave a trace. I found out that my email hoster doesn’t provide easy access to the IP addresses that access the mailbox. I couldn’t think of any other traces at the time, but next time I might check if my IP address is visible on some IRC server.</p>
<p>As a last resort I remembered that my ISP usually gives me quite similar IP addresses, so I used whois on yesterday’s IP address to see how large the subnet is. I got back a subnet of only 2¹⁴ addresses, which seemed reasonable enough to scan. I hope the ISP is not too mad at me for such a small port scan, but to reduce the impact I also told nmap to only scan the specific SSH port that I use.</p>
<p>After about 5 minutes I had found a single server responding on my custom SSH port, which was indeed my home server. After logging in and checking out the system I noticed the (usual) problem of cronie being unable to spawn new processes after a glibc update. So I simply restarted cronie to fix the problem.</p>
<p>I still wish that Arch Linux will one day tell me which running executables to restart after a system upgrade, as SLES does, to prevent problems like this in the future. I think the general approach is just to check which process references deleted shared libraries. Something like this, <a href="https://locallost.net/?p=233">similar to here</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ps axh <span class="nt">-o</span> pid | <span class="k">while </span><span class="nb">read </span>PID<span class="p">;</span> <span class="k">do </span><span class="nb">grep</span> <span class="s2">".so"</span> /proc/<span class="nv">$PID</span>/maps | <span class="nb">grep</span> <span class="s2">"(deleted)"</span> <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$PID</span><span class="s2">"</span> <span class="o">&&</span> <span class="nb">sed</span> <span class="nt">-e</span> <span class="s2">"s/</span><span class="se">\x</span><span class="s2">00/ /g"</span> < /proc/<span class="nv">$PID</span>/cmdline <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></figure>
<p>Update: <a href="https://github.com/tylerjl/overdue/">overdue</a> is a pacman post-transaction hook that lists daemons that still reference old shared libraries.</p>
Chinese Traffic to time.gif2019-05-21T00:00:00+02:00https://hookrace.net/blog/chinese-traffic-time.gif
<p>Nearly two years ago <a href="/blog/time.gif/">I posted</a> this endless GIF that always shows the current time in UTC:</p>
<!--more-->
<p><img src="/time.gif" alt="time.gif (reload if it fails)" /></p>
<p>Now looking at my <a href="https://goaccess.io/">GoAccess</a> dashboard I can see that it is picking up in popularity rather suddenly:</p>
<p><img src="/public/chinese-traffic/goaccess.png" alt="GoAccess excerpt" /></p>
<p>But strangely I can’t find anything about time.gif being linked on the web. So this might just be an attempted Denial of Service (DoS) attack? At least that would be something I am familiar with from the <a href="https://ddnet.org/">DDNet</a> direction, but it’s certainly strange on HookRace. But instead of simply shutting down time.gif I decided to try to find out who is accessing it and whether I can keep the server up.</p>
<p>Let’s look into the <a href="https://www.nginx.com/">nginx</a> logs, since I use nginx to proxy the requests to the Haskell program. There I see about 40 new requests per second looking like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>hookrace.net 123.185.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 3335 "XXX" "Mozilla/5.0 (Linux; Android 8.1.0; V1818A Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044681 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 8.055
hookrace.net 111.62.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 32061 "XXX" "Mozilla/5.0 (Linux; Android 5.1; OPPO A59s Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044704 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043B) Process/tools NetType/WIFI Language/zh_CN" 89.256
hookrace.net 111.29.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 543830 "XXX" "Mozilla/5.0 (Linux; Android 7.1.1; OPPO R11 Build/NMF26X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044704 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 1540.238
hookrace.net 112.2.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 172102 "XXX" "Mozilla/5.0 (Linux; Android 8.1.0; V1816A Build/OPM1.171019.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044611 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 492.600
hookrace.net 123.13.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 1275 "XXX" "Mozilla/5.0 (Linux; Android 9; LYA-AL00 Build/HUAWEILYA-AL00L; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.126 MQQBrowser/6.2 TBS/044704 Mobile Safari/537.36 MMWEBID/XXX MicroMessenger/7.0.4.1420(0x2700043A) Process/tools NetType/WIFI Language/zh_CN" 1.888
hookrace.net 117.91.XXX.XXX - - [21/May/2019:21:21:27 +0200] "GET /time.gif HTTP/2.0" 200 4684 "XXX" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/16D57 MicroMessenger/7.0.3(0x17000321) NetType/WIFI Language/zh_CN" 12.123
</code></pre></div></div>
<p>I checked a few IP addresses and they were all in mobile networks, not data centers. The user agent containing MicroMessenger and MQQBrowser indicates that the source of the traffic are WeChat and/or QQ, popular chinese chat apps.</p>
<h2 id="quantifying-the-traffic">Quantifying the Traffic</h2>
<p>For reference, the system I’m running on is a simple <a href="https://www.debian.org/">Debian</a> based VPS with 2 threads and 2 GB of RAM that also functions as the main server for <a href="https://ddnet.org/">DDNet’s website</a>, database and my HookRace blog.</p>
<p>I already had to do some scaling when posting the <a href="/blog/time.gif/">initial blog post</a> on <a href="https://news.ycombinator.com/item?id=14996715">Hacker News</a>, optimizing the Haskell application itself to use LZW encoding in the GIF frames, to properly clean up connections to prevent any memory leaks and disable buffering in nginx’s config.</p>
<p>But the current level of traffic is on a different scale with 2.4 million hits on time.gif in the last 23 hours (30 hits per second) resulting in 113 GB of data being transferred. And many of those connections don’t finish quickly, instead they linger for seconds, minutes or even hours.</p>
<p>Using <code class="language-plaintext highlighter-rouge">lsof -i | grep Time | wc -l</code> I can see that there are about 6000 people downloading the GIF at peak times, causing up to 30 Mbit/s of outgoing traffic with 7000 packets/second incoming and the same number outgoing. The <a href="https://ddnet.org/stats/server/">DDNet server statistics</a> lets me monitor this nicely (<a href="/blog/server-statistics/">related blog article</a>):</p>
<p>Network
<img src="/public/chinese-traffic/ddnet-network.png" alt="Network Traffic" />
Packets
<img src="/public/chinese-traffic/ddnet-packets.png" alt="Network Packets" />
CPU
<img src="/public/chinese-traffic/ddnet-cpu.png" alt="CPU Usage" /></p>
<h2 id="keeping-up-with-the-traffic">Keeping Up with the Traffic</h2>
<p>Regenerating the <a href="https://ddnet.org/ranks/">ranks pages of DDNet</a> usually causes the main CPU load on the server, which can be seen in the above CPU graph as spikes. This task is already set to only run when the server is below a specified load, so that more essential tasks have priority.</p>
<p>The first new problem was nginx running into a limit of 768 worker_connections:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2019/05/20 20:41:30 [alert] 761#761: *3828093 768 worker_connections are not enough while connecting to upstream, client: 49.114.XXX.XXX, server: hookrace.net, request: "GET /time.gif HTTP/2.0", upstream: "http://127.0.0.1:5002/", host: "hookrace.net", referrer: "XXX"
</code></pre></div></div>
<p>Luckily that is easily fixed in <code class="language-plaintext highlighter-rouge">/etc/nginx/nginx.conf</code> by increasing the number of worker_connections to keep alive, each of which is handling one of the long-lasting time.gif requests:</p>
<figure class="highlight"><pre><code class="language-nginx" data-lang="nginx"><span class="k">events</span> <span class="p">{</span>
<span class="kn">worker_connections</span> <span class="mi">20000</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>and <code class="language-plaintext highlighter-rouge">systemctl reload nginx</code>. No downtime required since nginx will start new worker processes to handle new requests while keeping the old ones alive for a time to keep handling existing connections.</p>
<p>Unfortunately that fix only lasted a few hours until the next problem appeared:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2019/05/20 23:09:21 [alert] 15188#15188: *4041619 socket() failed (24: Too many open files) while connecting to upstream, client: 27.207.XXX.XXX, server: hookrace.net, request: "GET /time.gif HTTP/2.0", upstream: "http://127.0.0.1:5002/", host: "hookrace.net", referrer: "XXX"
</code></pre></div></div>
<p>Increasing the limits in <code class="language-plaintext highlighter-rouge">/etc/security/limits.conf</code> for the nginx user fixes this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#<domain> <type> <item> <value>
nginx soft nofile 1048576
nginx hard nofile 1048576
</code></pre></div></div>
<p>The value of <code class="language-plaintext highlighter-rouge">1048576</code> is chosen since it’s the value set in <code class="language-plaintext highlighter-rouge">sysctl fs.file-max</code> and it should be good enough for now.</p>
<p>Next I noticed that the server was running out of memory with both the Haskell application and nginx having to keep track of so many connections at once. For now I increased the swap size on the fly to keep some less commonly used stuff there using <code class="language-plaintext highlighter-rouge">dd if=/dev/zero of=/var/swap bs=1M count=5000 && mkswap /var/swap && swapon /var/swap</code>.</p>
<p>When running out of memory I noticed that Python’s msgpack implementation <a href="https://github.com/msgpack/msgpack-python/issues/239">fails quite confusingly</a> when it runs OOM. So I had to add some fixes to the code creating the <a href="https://ddnet.org/ranks/">DDNet ranks pages</a> to handle this possibility.</p>
<p>The Linux Kernel’s TCP buffers ran out of memory next, complaining in dmesg:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[1638211.984805] TCP: out of memory -- consider tuning tcp_mem
</code></pre></div></div>
<p>So I increased them with a <code class="language-plaintext highlighter-rouge">net.ipv4.tcp_mem = 116730 155640 233460</code> in <code class="language-plaintext highlighter-rouge">/etc/sysctl.conf</code> and reloaded it with <code class="language-plaintext highlighter-rouge">sysctl -p</code>.</p>
<p>A limitation of my current approach is the number of ports nginx can open to proxy to the Haskell application. If that gets blown I’ll have to communicate to the application differently or simply redirect to the application directly instead of proxying it. That would also reduce the CPU load significantly, cutting out nginx which happens to be much more expensive than the Haskell application, probably because it’s also handling TLS.</p>
<h2 id="final-words">Final Words</h2>
<p>While it was fun to keep time.gif running in the face of this amount of traffic, I still haven’t answered the final question of where this traffic is coming from. It might be that lots of Chinese happen to be spreading time.gif on WeChat and QQ, but for that the traffic looks a bit too sterile. Has anyone seen similar traffic patterns and might know if they are real or some kind of botnet? Maybe someone has embedded traffic.gif on some WeChat-specific page. If anyone has a clue please drop me an email at <a href="mailto:dennis@felsing.org">dennis@felsing.org</a>.</p>
<p>The best guess so far is by Kolen:</p>
<blockquote>
<p>Hi,</p>
<p>I read your post and this is just my guess:</p>
<p>Chinese “WhatsApp” type of communication culture is very strange. They
are like spam emails in the old days. Often time some one might create
posts in picture, eg include warm message such as reminding each other to
put on more clothes as the weather is getting cold, etc.</p>
<p>My guess is then someone read your trick and thought that it is a good
idea to create some picture that always show the current time. Eg to
remind others that it is time to sleep or something.</p>
<p>And like email spams in the old days, people can share things like this
crazily, often by older people who don’t know much about technology.
(Just like how some tweet can go viral, messages like this could also go
viral within there networks. And unfortunately the viral thing we often
received are really rubbish like.)</p>
<p>yours,</p>
<p>Kolen</p>
</blockquote>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=19978393">Hacker News</a>.</p>
Linux Desktop Setup2019-01-15T00:00:00+01:00https://hookrace.net/blog/linux-desktop-setup
<p><a href="https://vectorified.com/ru-linux-desktop-setup">Russian Translation by Akhmad Karimov</a></p>
<p>My software setup has been surprisingly constant over the last decade, after a few years of experimentation since I initially switched to Linux in 2006. It might be interesting to look back in another 10 years and see what changed. A quick overview of what’s running as I’m writing this post:</p>
<!--more-->
<p><a href="/public/linux-desktop/htop.png"><img src="/public/linux-desktop/htop_small.png" alt="htop overview" /></a></p>
<h2 id="motivation">Motivation</h2>
<p>My software priorities are, in no specific order:</p>
<ul>
<li>Programs should run on my local system so that I’m in control of them, this excludes cloud solutions.</li>
<li>Programs should run in the terminal, so that they can be used consistently from anywhere, including weak computers or a phone.</li>
<li>Keyboard focused is nearly automatic by using terminal software. I prefer to use the mouse where it makes sense only, reaching for the mouse all the time during typing feels like a waste of time. Occasionally it took me an hour to notice that the mouse wasn’t even plugged in.</li>
<li>Ideally use fast and efficient software, I don’t like hearing the fan and feeling the room heat up. I can also keep running older hardware for much longer, my 10 year old Thinkpad x200s is still fine for all the software I use.</li>
<li>Be composable. I don’t want to do every step manually, instead automate more when it makes sense. This naturally favors the shell.</li>
</ul>
<h2 id="operating-systems">Operating Systems</h2>
<p>I had a hard start with Linux 12 years ago by removing Windows, armed with just the <a href="https://gentoo.org/">Gentoo Linux</a> installation CD and a printed manual to get a functioning Linux system. It took me a few days of compiling and tinkering, but in the end I felt like I had learnt a lot.</p>
<p>I haven’t looked back to Windows since then, but I switched to <a href="https://www.archlinux.org/">Arch Linux</a> on my laptop after having the fan fail from the constant compilation stress. Later I also switched all my other computers and private servers to Arch Linux. As a rolling release distribution you get package upgrades all the time, but the most important breakages are nicely reported in the <a href="https://www.archlinux.org/news/">Arch Linux News</a>.</p>
<p>One annoyance though is that Arch Linux removes the old kernel modules once you upgrade it. I usually notice that once I try plugging in a USB flash drive and the kernel fails to load the relevant module. Instead you’re supposed to reboot after each kernel upgrade. There are a few <a href="https://www.reddit.com/r/archlinux/comments/4zrsc3/keep_your_system_fully_functional_after_a_kernel/">hacks</a> around to get around the problem, but I haven’t been bothered enough to actually use them.</p>
<p>Similar problems happen with other programs, commonly Firefox, cron or Samba requiring a restart after an upgrade, but annoyingly not warning you that that’s the case. <a href="https://www.suse.com/">SUSE</a>, which I use at work, nicely warns about such cases.</p>
<p>For the <a href="https://ddnet.org/">DDNet</a> production servers I prefer <a href="https://www.debian.org/">Debian</a> over Arch Linux, so that I have a lower chance of breakage on each upgrade. For my firewall and router I used <a href="https://www.openbsd.org/">OpenBSD</a> for its clean system, documentation and great <a href="https://www.openbsd.org/faq/pf/">pf firewall</a>, but right now I don’t have a need for a separate router anymore.</p>
<h2 id="window-manager">Window Manager</h2>
<p>Since I started out with Gentoo I quickly noticed the huge compile time of KDE, which made it a no-go for me. I looked around for more minimal solutions, and used <a href="http://openbox.org/wiki/Main_Page">Openbox</a> and <a href="http://fluxbox.org/">Fluxbox</a> initially. At some point I jumped on the tiling window manager train in order to be more keyboard-focused and picked up <a href="https://dwm.suckless.org/">dwm</a> and <a href="https://awesomewm.org/">awesome</a> close to their initial releases.</p>
<p>In the end I settled on <a href="https://xmonad.org/">xmonad</a> thanks to its flexibility, extendability and being written and configured in pure <a href="https://www.haskell.org/">Haskell</a>, a great functional programming language. One example of this is that at home I run a single 40” 4K screen, but often split it up into four virtual screens, each displaying a workspace on which my windows are automatically arranged. Of course xmonad has a <a href="http://hackage.haskell.org/package/xmonad-contrib-0.15/docs/XMonad-Layout-LayoutScreens.html">module</a> for that.</p>
<p><a href="http://robm.github.io/dzen/">dzen</a> and <a href="https://github.com/brndnmtthws/conky">conky</a> function as a simple enough status bar for me. My entire conky config looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out_to_console yes
update_interval 1
total_run_times 0
TEXT
${downspeed eth0} ${upspeed eth0} | $cpu% ${loadavg 1} ${loadavg 2} ${loadavg 3} $mem/$memmax | ${time %F %T}
</code></pre></div></div>
<p>And gets piped straight into dzen2 with <code class="language-plaintext highlighter-rouge">conky | dzen2 -fn '-xos4-terminus-medium-r-normal-*-12-*-*-*-*-*-*-*' -bg '#000000' -fg '#ffffff' -p -e '' -x 1000 -w 920 -xs 1 -ta r</code>.</p>
<p>One important feature for me is to make the terminal emit a beep sound once a job is done. This is done simply by adding a <code class="language-plaintext highlighter-rouge">\a</code> character to the <code class="language-plaintext highlighter-rouge">PR_TITLEBAR</code> variable in zsh, which is shown whenever a job is done. Of course I disable the actual beep sound by blacklisting the <code class="language-plaintext highlighter-rouge">pcspkr</code> kernel module with <code class="language-plaintext highlighter-rouge">echo "blacklist pcspkr" > /etc/modprobe.d/nobeep.conf</code>. Instead the sound gets turned into an urgency by urxvt’s <code class="language-plaintext highlighter-rouge">URxvt.urgentOnBell: true</code> setting. Then xmonad has an urgency hook to capture this and I can automatically focus the currently urgent window with a key combination. In dzen I get the urgent windowspaces displayed with a nice and bright <code class="language-plaintext highlighter-rouge">#ff0000</code>.</p>
<p>The final result in all its glory on my Laptop:</p>
<p><a href="/public/linux-desktop/laptop.png"><img src="/public/linux-desktop/laptop_small.png" alt="Laptop screenshot" /></a></p>
<p>I hear that <a href="https://i3wm.org/">i3</a> has become quite popular in the last years, but it requires more manual window alignment instead of specifying automated methods to do it.</p>
<p>I realize that there are also terminal multiplexers like <a href="https://github.com/tmux/tmux/wiki">tmux</a>, but I still require a few graphical applications, so in the end I never used them productively.</p>
<h2 id="terminal-persistency">Terminal Persistency</h2>
<p>In order to keep terminals alive I use <a href="http://dtach.sourceforge.net/">dtach</a>, which is just the detach feature of screen. In order to make every terminal on my computer detachable I wrote a <a href="https://github.com/def-/tach/blob/master/tach">small wrapper script</a>. This means that even if I had to restart my X server I could keep all my terminals running just fine, both local and remote.</p>
<h2 id="shell--programming">Shell & Programming</h2>
<p>Instead of <a href="https://www.gnu.org/software/bash/">bash</a> I use <a href="http://www.zsh.org/">zsh</a> as my shell for its huge number of features.</p>
<p>As a terminal emulator I found <a href="http://software.schmorp.de/pkg/rxvt-unicode.html">urxvt</a> to be simple enough, support Unicode and 256 colors and has great performance. Another great feature is being able to run the urxvt client and daemon separately, so that even a large number of terminals barely takes up any memory (except for the scrollback buffer).</p>
<p>There is only one font that looks absolutely clean and perfect to me: <a href="http://terminus-font.sourceforge.net/">Terminus</a>. Since it’s a bitmap font everything is pixel perfect and renders extremely fast and at low CPU usage. In order to switch fonts on-demand in each terminal with <code class="language-plaintext highlighter-rouge">CTRL-WIN-[1-7]</code> my ~/.Xdefaults contains:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>URxvt.font: -xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*
dzen2.font: -xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*
URxvt.keysym.C-M-1: command:\033]50;-xos4-terminus-medium-r-normal-*-12-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-2: command:\033]50;-xos4-terminus-medium-r-normal-*-14-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-3: command:\033]50;-xos4-terminus-medium-r-normal-*-18-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-4: command:\033]50;-xos4-terminus-medium-r-normal-*-22-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-5: command:\033]50;-xos4-terminus-medium-r-normal-*-24-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-6: command:\033]50;-xos4-terminus-medium-r-normal-*-28-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-7: command:\033]50;-xos4-terminus-medium-r-normal-*-32-*-*-*-*-*-*-*\007
URxvt.keysym.C-M-n: command:\033]10;#ffffff\007\033]11;#000000\007\033]12;#ffffff\007\033]706;#00ffff\007\033]707;#ffff00\007
URxvt.keysym.C-M-b: command:\033]10;#000000\007\033]11;#ffffff\007\033]12;#000000\007\033]706;#0000ff\007\033]707;#ff0000\007
</code></pre></div></div>
<p>For programming and writing I use <a href="https://www.vim.org/">Vim</a> with syntax highlighting and <a href="http://ctags.sourceforge.net/">ctags</a> for indexing, as well as a few terminal windows with grep, sed and the other usual suspects for search and manipulation. This is probably not at the same level of comfort as an IDE, but allows me more automation.</p>
<p>One problem with Vim is that you get so used to its key mappings that you’ll want to use them everywhere.</p>
<p><a href="https://www.python.org/">Python</a> and <a href="https://nim-lang.org/">Nim</a> do well as scripting languages where the shell is not powerful enough.</p>
<h2 id="system-monitoring">System Monitoring</h2>
<p><a href="https://hisham.hm/htop/">htop</a> works great for getting a quick overview of what the software is currently doing. <a href="http://lm-sensors.org/">lm_sensors</a> allows monitoring the hardware temperatures, fans and voltages. <a href="https://01.org/powertop/">powertop</a> is a great little tool by Intel to find power savings. <a href="https://dev.yorhel.nl/ncdu">ncdu</a> lets you analyze disk usage interactively.</p>
<p><a href="https://nmap.org/">nmap</a>, iptraf-ng, <a href="https://www.tcpdump.org/">tcpdump</a> and <a href="https://www.wireshark.org/">Wireshark</a> are essential tools for analyzing network problems.</p>
<p>There are of course many more great tools.</p>
<h2 id="mails--synchronization">Mails & Synchronization</h2>
<p>On my home server I have a <a href="http://www.fetchmail.info/">fetchmail</a> daemon running for each email acccount that I have. Fetchmail just retrieves the incoming emails and invokes <a href="http://www.procmail.org/">procmail</a>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/sh</span>
<span class="k">for </span>i <span class="k">in</span> /home/deen/.fetchmail/<span class="k">*</span><span class="p">;</span> <span class="k">do
</span><span class="nv">FETCHMAILHOME</span><span class="o">=</span><span class="nv">$i</span> /usr/bin/fetchmail <span class="nt">-m</span> <span class="s1">'procmail -d %T'</span> <span class="nt">-d</span> 60
<span class="k">done</span></code></pre></figure>
<p>The configuration is as simple as it could be and waits for the server to inform us of fresh emails:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>poll imap.1und1.de protocol imap timeout 120 user "dennis@felsing.org" password "XXX" folders INBOX keep ssl idle
</code></pre></div></div>
<p>My <code class="language-plaintext highlighter-rouge">.procmailrc</code> config contains a few rules to backup all mails and sort them into the correct directories, for example based on the mailing list id or from field in the mail header:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MAILDIR=/home/deen/shared/Maildir
LOGFILE=$HOME/.procmaillog
LOGABSTRACT=no
VERBOSE=off
FORMAIL=/usr/bin/formail
NL="
"
:0wc
* ! ? test -d /media/mailarchive/`date +%Y`
| mkdir -p /media/mailarchive/`date +%Y`
# Make backups of all mail received in format YYYY/YYYY-MM
:0c
/media/mailarchive/`date +%Y`/`date +%Y-%m`
:0
* ^From: .*(.*@.*.kit.edu|.*@.*.uka.de|.*@.*.uni-karlsruhe.de)
$MAILDIR/.uni/
:0
* ^list-Id:.*lists.kit.edu
$MAILDIR/.uni-ml/
[...]
</code></pre></div></div>
<p>To send emails I use <a href="https://marlam.de/msmtp/">msmtp</a>, which is also great to configure:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>account default
host smtp.1und1.de
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on
from dennis@felsing.org
user dennis@felsing.org
password XXX
[...]
</code></pre></div></div>
<p>But so far the emails are still on the server. My documents are all stored in a directory that I synchronize between all computers using <a href="https://www.cis.upenn.edu/~bcpierce/unison/">Unison</a>. Think of Unison as a bidirectional interactive <a href="https://rsync.samba.org/">rsync</a>. My emails are part of this documents directory and thus they end up on my desktop computers.</p>
<p>This also means that while the emails reach my server immediately, I only fetch them on deman instead of getting instant notifications when an email comes in.</p>
<p>From there I read the mails with <a href="http://www.mutt.org/">mutt</a>, using the sidebar plugin to display my mail directories. The <code class="language-plaintext highlighter-rouge">/etc/mailcap</code> file is essential to display non-plaintext mails containing HTML, Word or PDF:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>text/html;w3m -I %{charset} -T text/html; copiousoutput
application/msword; antiword %s; copiousoutput
application/pdf; pdftotext -layout /dev/stdin -; copiousoutput
</code></pre></div></div>
<h2 id="news--communication">News & Communication</h2>
<p><a href="https://newsboat.org/">Newsboat</a> is a nice little RSS/Atom feed reader in the terminal. I have it running on the server in a <code class="language-plaintext highlighter-rouge">tach</code> session with about 150 feeds. Filtering feeds locally is also possible, for example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ignore-article "https://forum.ddnet.org/feed.php" "title =~ \"Map Testing •\" or title =~ \"Old maps •\" or title =~ \"Map Bugs •\" or title =~ \"Archive •\" or title =~ \"Waiting for mapper •\" or title =~ \"Other mods •\" or title =~ \"Fixes •\""
</code></pre></div></div>
<p>I use <a href="https://irssi.org/">Irssi</a> the same way for communication via IRC.</p>
<h2 id="calendar">Calendar</h2>
<p><a href="https://www.roaringpenguin.com/products/remind">remind</a> is a calendar that can be used from the command line. Setting new reminders is done by editing the <code class="language-plaintext highlighter-rouge">rem</code> files:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># One time events
REM 2019-01-20 +90 Flight to China %b
# Recurring Holidays
REM 1 May +90 Holiday "Tag der Arbeit" %b
REM [trigger(easterdate(year(today()))-2)] +90 Holiday "Karfreitag" %b
# Time Change
REM Nov Sunday 1 --7 +90 Time Change (03:00 -> 02:00) %b
REM Apr Sunday 1 --7 +90 Time Change (02:00 -> 03:00) %b
# Birthdays
FSET birthday(x) "'s " + ord(year(trigdate())-x) + " birthday is %b"
REM 16 Apr +90 MSG Andreas[birthday(1994)]
# Sun
SET $LatDeg 49
SET $LatMin 19
SET $LatSec 49
SET $LongDeg -8
SET $LongMin -40
SET $LongSec -24
MSG Sun from [sunrise(trigdate())] to [sunset(trigdate())]
[...]
</code></pre></div></div>
<p>Unfortunately there is no Chinese Lunar calendar function in remind yet, so Chinese holidays can’t be calculated easily.</p>
<p>I use two aliases for remind:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rem -m -b1 -q -g
</code></pre></div></div>
<p>to see a list of the next events in chronological order and</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rem -m -b1 -q -cuc12 -w$(($(tput cols)+1)) | sed -e "s/\f//g" | less
</code></pre></div></div>
<p>to show a calendar fitting just the width of my terminal:</p>
<p><img src="/public/linux-desktop/remcal.png" alt="remcal" /></p>
<h2 id="dictionary">Dictionary</h2>
<p><a href="https://github.com/tsdh/rdictcc">rdictcc</a> is a little known dictionary tool that uses the excellent dictionary files from <a href="https://www.dict.cc/">dict.cc</a> and turns them into a local database:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ rdictcc rasch
====================[ A => B ]====================
rasch:
- apace
- brisk [speedy]
- cursory
- in a timely manner
- quick
- quickly
- rapid
- rapidly
- sharpish [Br.] [coll.]
- speedily
- speedy
- swift
- swiftly
rasch [gehen]:
- smartly [quickly]
Rasch {n} [Zittergras-Segge]:
- Alpine grass [Carex brizoides]
- quaking grass sedge [Carex brizoides]
Rasch {m} [regional] [Putzrasch]:
- scouring pad
====================[ B => A ]====================
Rasch model:
- Rasch-Modell {n}
</code></pre></div></div>
<h2 id="writing-and-reading">Writing and Reading</h2>
<p>I have a simple todo file containing my tasks, that is basically always sitting open in a Vim session. For work I also use the todo file as a “done” file so that I can later check what tasks I finished on each day.</p>
<p>For writing documents, letters and presentations I use <a href="https://www.latex-project.org/">LaTeX</a> for its superior typesetting. A simple letter in German format can be set like this for example:</p>
<figure class="highlight"><pre><code class="language-latex" data-lang="latex"><span class="k">\documentclass</span><span class="na">[paper = a4, fromalign = right]</span><span class="p">{</span>scrlttr2<span class="p">}</span>
<span class="k">\usepackage</span><span class="p">{</span>german<span class="p">}</span>
<span class="k">\usepackage</span><span class="p">{</span>eurosym<span class="p">}</span>
<span class="k">\usepackage</span><span class="na">[utf8]</span><span class="p">{</span>inputenc<span class="p">}</span>
<span class="k">\setlength</span><span class="p">{</span><span class="k">\parskip</span><span class="p">}{</span>6pt<span class="p">}</span>
<span class="k">\setlength</span><span class="p">{</span><span class="k">\parindent</span><span class="p">}{</span>0pt<span class="p">}</span>
<span class="k">\setkomavar</span><span class="p">{</span>fromname<span class="p">}{</span>Dennis Felsing<span class="p">}</span>
<span class="k">\setkomavar</span><span class="p">{</span>fromaddress<span class="p">}{</span>Meine Str. 1<span class="k">\\</span>69181 Leimen<span class="p">}</span>
<span class="k">\setkomavar</span><span class="p">{</span>subject<span class="p">}{</span>Titel<span class="p">}</span>
<span class="k">\setkomavar*</span><span class="p">{</span>enclseparator<span class="p">}{</span>Anlagen<span class="p">}</span>
<span class="k">\makeatletter</span>
<span class="k">\@</span>setplength<span class="p">{</span>refvpos<span class="p">}{</span>89mm<span class="p">}</span>
<span class="k">\makeatother</span>
<span class="nt">\begin{document}</span>
<span class="nt">\begin{letter}</span> <span class="p">{</span>Herr Soundso<span class="k">\\</span>Deine Str. 2<span class="k">\\</span>69121 Heidelberg<span class="p">}</span>
<span class="k">\opening</span><span class="p">{</span>Sehr geehrter Herr Soundso,<span class="p">}</span>
Sie haben bei mir seit dem Bla Bla Bla.
Ich fordere Sie hiermit zu Bla Bla Bla auf.
<span class="k">\closing</span><span class="p">{</span>Mit freundlichen Grüßen<span class="p">}</span>
<span class="nt">\end{letter}</span>
<span class="nt">\end{document}</span></code></pre></figure>
<p>Further example documents and presentations can be found over at <a href="https://dennis.felsing.org/research/">my private site</a>.</p>
<p>To read PDFs <a href="https://pwmt.org/projects/zathura/">Zathura</a> is fast, has Vim-like controls and even supports two different PDF backends: Poppler and MuPDF. <a href="https://wiki.gnome.org/Apps/Evince">Evince</a> on the other hand is more full-featured for the cases where I encounter documents that Zathura doesn’t like.</p>
<h2 id="graphical-editing">Graphical Editing</h2>
<p><a href="https://www.gimp.org/">GIMP</a> and <a href="https://inkscape.org/">Inkscape</a> are easy choices for photo editing and interactive vector graphics respectively.</p>
<p>In some cases <a href="https://imagemagick.org/Usage/">Imagemagick</a> is good enough though and can be used straight from the command line and thus automated to edit images. Similarly <a href="https://www.graphviz.org/">Graphviz</a> and <a href="https://sourceforge.net/projects/pgf/">TikZ</a> can be used to draw graphs and other diagrams.</p>
<h2 id="web-browsing">Web Browsing</h2>
<p>As a web browser I’ve always used <a href="https://www.mozilla.org/en-US/firefox/new/">Firefox</a> for its extensibility and low resource usage compared to Chrome.</p>
<p>Unfortunately the <a href="https://github.com/5digits/dactyl">Pentadactyl</a> extension development stopped after Firefox switched to Chrome-style extensions entirely, so I don’t have satisfying Vim-like controls in my browser anymore.</p>
<h2 id="media-players">Media Players</h2>
<p><a href="https://mpv.io/">mpv</a> with hardware decoding allows watching videos at 5% CPU load using the <code class="language-plaintext highlighter-rouge">vo=gpu</code> and <code class="language-plaintext highlighter-rouge">hwdec=vaapi</code> config settings. <code class="language-plaintext highlighter-rouge">audio-channels=2</code> in mpv seems to give me clearer downmixing to my stereo speakers / headphones than what PulseAudio does by default. A great little feature is exiting with <code class="language-plaintext highlighter-rouge">Shift-Q</code> instead of just <code class="language-plaintext highlighter-rouge">Q</code> to save the playback location. When watching with someone with another native tongue you can use <code class="language-plaintext highlighter-rouge">--secondary-sid=</code> to show two subtitles at once, the primary at the bottom, the secondary at the top of the screen</p>
<p>My wirelss mouse can easily be made into a remote control with mpv with a small <code class="language-plaintext highlighter-rouge">~/.config/mpv/input.conf</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MOUSE_BTN5 run "mixer" "pcm" "-2"
MOUSE_BTN6 run "mixer" "pcm" "+2"
MOUSE_BTN1 cycle sub-visibility
MOUSE_BTN7 add chapter -1
MOUSE_BTN8 add chapter 1
</code></pre></div></div>
<p><a href="https://rg3.github.io/youtube-dl/">youtube-dl</a> works great for watching videos hosted online, best quality can be achieved with <code class="language-plaintext highlighter-rouge">-f bestvideo+bestaudio/best --all-subs --embed-subs</code>.</p>
<p>As a music player <a href="http://moc.daper.net/">MOC</a> hasn’t been actively developed for a while, but it’s still a simple player that plays every format conceivable, including the strangest Chiptune formats. In the AUR there is a <a href="https://aur.archlinux.org/packages/moc-pulse/">patch</a> adding PulseAudio support as well. Even with the CPU clocked down to 800 MHz MOC barely uses 1-2% of a single CPU core.</p>
<p><img src="/public/linux-desktop/moc.png" alt="moc" /></p>
<p>My music collection sits on my home server so that I can access it from anywhere. It is mounted using <a href="https://github.com/libfuse/sshfs">SSHFS</a> and automount in the <code class="language-plaintext highlighter-rouge">/etc/fstab/</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@server:/media/media /mnt/media fuse.sshfs noauto,x-systemd.automount,idmap=user,IdentityFile=/root/.ssh/id_rsa,allow_other,reconnect 0 0
</code></pre></div></div>
<h2 id="cross-platform-building">Cross-Platform Building</h2>
<p>Linux is great to build packages for any major operating system except Linux itself! In the beginning I used <a href="https://www.qemu.org/">QEMU</a> to with an old Debian, Windows and Mac OS X VM to build for these platforms.</p>
<p>Nowadays I switched to using chroot for the old Debian distribution (for maximum Linux compatibility), <a href="http://www.mingw.org/">MinGW</a> to cross-compile for Windows and <a href="https://github.com/tpoechtrager/osxcross">OSXCross</a> to cross-compile for Mac OS X.</p>
<p>The script used to <a href="https://github.com/ddnet/ddnet-scripts/blob/master/ddnet-release.sh">build DDNet</a> as well as the <a href="https://github.com/ddnet/ddnet-scripts/blob/master/ddnet-lib-update.sh">instructions for updating library builds</a> are based on this.</p>
<h2 id="backups">Backups</h2>
<p>As usual, we nearly forgot about backups. Even if this is the last chapter, it should not be an afterthought.</p>
<p>I wrote <a href="https://github.com/def-/rrb/blob/master/rrb">rrb</a> (reverse rsync backup) 10 years ago to wrap rsync so that I only need to give the backup server root SSH rights to the computers that it is backing up. Surprisingly rrb needed 0 changes in the last 10 years, even though I kept using it the entire time.</p>
<p>The backups are stored straight on the filesystem. Incremental backups are implemented using hard links (<code class="language-plaintext highlighter-rouge">--link-dest</code>). A simple <a href="https://github.com/def-/rrb/blob/master/config.example">config</a> defines how long backups are kept, which defaults to:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">KEEP_RULES</span><span class="o">=(</span> <span class="se">\</span>
7 7 <span class="se">\ </span><span class="c"># One backup a day for the last 7 days</span>
31 8 <span class="se">\ </span><span class="c"># 8 more backups for the last month</span>
365 11 <span class="se">\ </span><span class="c"># 11 more backups for the last year</span>
1825 4 <span class="se">\ </span><span class="c"># 4 more backups for the last 5 years</span>
<span class="o">)</span></code></pre></figure>
<p>Since some of my computers don’t have a static IP / DNS entry and I still want to back them up using rrb I use a reverse SSH tunnel (as a systemd service) for them:</p>
<figure class="highlight"><pre><code class="language-cfg" data-lang="cfg">[Unit]
Description=Reverse SSH Tunnel
After=network.target
[Service]
ExecStart=/usr/bin/ssh -N -R 27276:localhost:22 -o "ExitOnForwardFailure yes" server
KillMode=process
Restart=always
[Install]
WantedBy=multi-user.target</code></pre></figure>
<p>Now the server can reach the client through <code class="language-plaintext highlighter-rouge">ssh -p 27276 localhost</code> while the tunnel is running to perform the backup, or in <code class="language-plaintext highlighter-rouge">.ssh/config</code> format:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host cr-remote
HostName localhost
Port 27276
</code></pre></div></div>
<p>While talking about SSH hacks, sometimes a server is not easily reachable thanks to some bad routing. In that case you can route the SSH connection through another server to get better routing, in this case going through the USA to reach my Chinese server which had not been reliably reachable from Germany for a few weeks:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host chn.ddnet.org
ProxyCommand ssh -q usa.ddnet.org nc -q0 chn.ddnet.org 22
Port 22
</code></pre></div></div>
<h2 id="final-remarks">Final Remarks</h2>
<p>Thanks for reading my random collection of tools. I probably forgot many programs that I use so naturally every day that I don’t even think about them anymore. Let’s see how stable my software setup stays in the next years. If you have any questions, feel free to get in touch with me at <a href="mailto:dennis@felsing.org">dennis@felsing.org</a>.</p>
<p>Comments on <a href="https://news.ycombinator.com/item?id=19253072">Hacker News</a>.</p>
Theme Switcher in Nginx2019-01-09T00:00:00+01:00https://hookrace.net/blog/theme-switcher-nginx
<p>Some people really liked the dark <a href="https://ddnet.org/">DDNet</a> theme for Halloween by <a href="">Soreu</a>, so we decided to keep it possible to use the default bright or the dark theme.</p>
<p>Thanks to xse we got a <a href="https://github.com/ddnet/ddnet-web/pull/69">JavaScript based theme switcher</a>. After some improvements I finally I switched it away from JavaScript entirely and finally am also using it on this blog with an OLED friendly dark theme.</p>
<!--more-->
<p>The final result is quite simple, and only requires static HTML, a cookie for the theme and Nginx for setting and returning the correct CSS file for it. You can try it out by clicking on the <a href="/switch-theme/">Switch Theme</a> button at the top of this page, which just redirects you to <code class="language-plaintext highlighter-rouge">/switch-theme/</code>.</p>
<p>Doing the CSS decision server-side instead of in JavaScript has the advantage that you don’t get any flicker on rendering, no matter what theme you chose.</p>
<p>The entire nginx logic is:</p>
<figure class="highlight"><pre><code class="language-nginx" data-lang="nginx"><span class="k">location</span> <span class="n">/switch-theme/</span> <span class="p">{</span>
<span class="kn">if</span> <span class="s">(</span><span class="nv">$http_cookie</span> <span class="p">~</span><span class="sr">*</span> <span class="s">"user_theme=/public/css-dark.css")</span> <span class="p">{</span>
<span class="kn">add_header</span> <span class="s">Set-Cookie</span> <span class="s">"user_theme=/public/css-empty.css</span><span class="p">;</span><span class="kn">path=/</span><span class="p">;</span><span class="kn">domain=hookrace.net"</span><span class="p">;</span>
<span class="kn">return</span> <span class="mi">302</span> <span class="nv">$http_referer</span><span class="p">;</span>
<span class="p">}</span>
<span class="kn">add_header</span> <span class="s">Set-Cookie</span> <span class="s">"user_theme=/public/css-dark.css</span><span class="p">;</span><span class="kn">path=/</span><span class="p">;</span><span class="kn">domain=hookrace.net"</span><span class="p">;</span>
<span class="kn">return</span> <span class="mi">302</span> <span class="nv">$http_referer</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">location</span> <span class="n">/public/css-dark.css</span> <span class="p">{</span>
<span class="kn">if</span> <span class="s">(</span><span class="nv">$http_cookie</span> <span class="p">~</span><span class="sr">*</span> <span class="s">"user_theme=/public/css-dark.css")</span> <span class="p">{</span>
<span class="kn">return</span> <span class="mi">302</span> <span class="n">/public/css/css-dark.css</span><span class="p">;</span>
<span class="p">}</span>
<span class="kn">return</span> <span class="mi">302</span> <span class="n">/public/css/css-empty.css</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>Note that the <code class="language-plaintext highlighter-rouge">/switch-theme/</code> location sends the user back to the <code class="language-plaintext highlighter-rouge">$http_referer</code>. <code class="language-plaintext highlighter-rouge">/public/css-dark.css</code> is statically included on each page, but what it does depends on the cookie value.</p>
<p>The <code class="language-plaintext highlighter-rouge">302 Moved Temporarily</code> response makes the client send another request, adding another roundtrip to the page rendering. If the CSS is small enough you can just pull it directly into the Nginx config:</p>
<figure class="highlight"><pre><code class="language-nginx" data-lang="nginx"><span class="k">location</span> <span class="n">/public/css-dark.css</span> <span class="p">{</span>
<span class="kn">if</span> <span class="s">(</span><span class="nv">$http_cookie</span> <span class="p">~</span><span class="sr">*</span> <span class="s">"user_theme=/public/css-dark.css")</span> <span class="p">{</span>
<span class="kn">return</span> <span class="mi">200</span> <span class="s">"body</span> <span class="p">{</span><span class="kn">color:</span> <span class="c1">#fff; background-color: #000;} .page-title, .post-title, .post-title a, h1, h2, h3, h4, h5, h6 {color: #ccc;} .masthead-title a {color: #bbb;}";</span>
<span class="err">}</span>
<span class="s">return</span> <span class="mi">200</span> <span class="s">""</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>One side effect of this entire approach is that the CSS file can’t be cached anymore though, otherwise we’d end up showing the old theme when it gets switched. I’m sure this could be optimized somehow to tell the browser to only fetch the CSS file again after the theme was switched. But for my site the current performance is good enough.</p>
<p>Also note that <a href="https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/">If Is Evil</a> in Nginx, so if you abuse features like this you might run into trouble.</p>
One year of cycling to work2018-02-20T00:00:00+01:00https://hookrace.net/blog/cycling-to-work
<p>Exactly on this day, one year ago, I came back from a one month long trip to
Taiwan, went straight to work from the airport and immediately moved into a new
apartment after work. Since then I have cycled to work nearly every day.</p>
<!--more-->
<iframe allowfullscreen="" class="osm-map" src="https://umap.openstreetmap.fr/en/map/one-year-of-cycling-to-work_199573?scaleControl=true&miniMap=false&scrollWheelZoom=true&zoomControl=false&allowEdit=false&moreControl=false&searchControl=false&tilelayersControl=false&embedControl=false&datalayersControl=false&onLoadPanel=undefined&captionBar=false"></iframe>
<p><a href="https://umap.openstreetmap.fr/en/map/one-year-of-cycling-to-work_199573#13/49.3166/8.6605">Fullscreen</a> (map made with <a href="https://umap.openstreetmap.fr/en/">umap</a>, routes made with <a href="https://www.graphhopper.com/">GraphHopper</a>)</p>
<h2 id="alternatives-and-motivation">Alternatives and motivation</h2>
<p>Previously I lived in Heidelberg, 15 km from work and usually took the train,
only cycling to work about once a week. But my new apartment was significantly
closer to work, only 6 km, which takes just 15-20 minutes to cycle, so it
became much easier to use the bicycle to get to work every day.</p>
<p>Taking public transportation instead is possible, and takes about 35-40
minutes. I only do that when I want to go to the city straight after work
instead of going home first. For me personally public transportation is more
useful in combination with the bicycle on longer trips.</p>
<p>Sometimes I take public transportation to work and just walk back in the
evening, which can be great for listening to audiobooks.</p>
<p>For about a month I had borrowed a car and sometimes used it to go to work.
Without traffic the driving would take 12 minutes, but with traffic it’s
usually 15-25 minutes. Additionally I have to get a parking spot and walk
further to the office, so using a car doesn’t save time compared to cycling.</p>
<p><a href="/public/cycling/IMG_20170414_091754.jpg"><img src="/public/cycling/IMG_20170414_091754_small.jpg" alt="Public transportation" /></a></p>
<h2 id="advantages">Advantages</h2>
<p>I remember feeling tired when getting to work early in the morning using public
transportation. After all you just keep sitting and standing, never really jump
starting the body. While cycling to work I certainly wake up and feel more
energetic and focussed when I arrive.</p>
<p>Having a clear separation of work and private life is important for me, both by
using different spaces for each as well as having a strict time separation. So
after I get out of work and get onto my bicycle I will still be having
work-related thoughts circling in my head. But while cycling I might take a
quick stop to write down any insights that I have or tasks that I forgot to
finish. Then I can finally shut down and clear my brain from work by enjoying
the ride, the views and the physical exercise.</p>
<p><a href="/public/cycling/IMG_20170409_184910-PANO.jpg"><img src="/public/cycling/IMG_20170409_184910-PANO_small.jpg" alt="Stream" /></a></p>
<p>I find the simplicity of cycling for locomotion beautiful. I never have to be
stuck in a traffic jam, feeling stressed by moving at a snail’s pace. Even if
cycling is slower the feeling of making progress is stronger. I never have to
be waiting for a late train, a bus that left too early or be stuck inside of a
train while there are problems with the power line. In the worst case I can
pick up my bicycle and walk around some obstacle or fix a flat tire in a few
minutes. Being outside in nature, at least for half an hour every day, feels
good.</p>
<p>Combined with the gym at work I feel like I’m getting enough sports to be
reasonably fit without having to spend much time on it. I regularly get home
from work at 17:00 with all work and sports done and can enjoy the rest of the
day without any worry.</p>
<p>When I’m using the bicycle to get to work already, I feel like the barrier to
combining this with larger trips is lowered. I occasionally do 40-50 km cycling
tours before or after work, depending on the weather and my mood.</p>
<p><a href="/public/cycling/IMG_20170614_172048.jpg"><img src="/public/cycling/IMG_20170614_172048_small.jpg" alt="Longer trip" /></a></p>
<h2 id="accommodation">Accommodation</h2>
<p>At home my bicycle is locked into a room in the basement, making it reasonably
secure and still easily accessible. At work we have roofed-over parking spaces
for the bicycles with robust metal bars for locking. Some of the cycle racks
even have two floors with an easy mechanism for getting your bicycle down from
there. When I don’t need my lock I will just leave it there after work for the
next day.</p>
<p>Multiple buildings at work have special showers for cyclists. But to be honest
I don’t use them after my usual route to work, for three reasons:</p>
<ol>
<li>It is cooler in the morning.</li>
<li>The distance is short enough.</li>
<li>I cycle slower when going to work in the morning than when going back.</li>
</ol>
<p>Especially in the summer it is nice to do a two hour cycling trip at 5 am and
go straight to work afterwards. Then the showers are a godsend of course.</p>
<p><a href="/public/cycling/IMG_20170404_073617.jpg"><img src="/public/cycling/IMG_20170404_073617_small.jpg" alt="Morning fog" /></a></p>
<h2 id="the-journey-is-the-reward">The journey is the reward</h2>
<p>Close to my home I have to cross a red light, which is also where I had the
only accident in this year. A car driver ignored my right of way and so I had
to break hard and ended up sitting on their hood. But luckily that didn’t
result in any injuries or damages. (Edit: I have never had an accident
involving a car before, so this is probably an outlier.)</p>
<p>After exiting the industrial area I am riding exclusively on small field roads.
Especially in the morning this makes for nice sunrises with fog on parts of the
fields. Along the way there is also a fishing lake.</p>
<p><a href="/public/cycling/IMG_20171018_170639.jpg"><img src="/public/cycling/IMG_20171018_170639_small.jpg" alt="Fishing lake" /></a></p>
<p>The way tunnels below train tracks, runs over a stream, tunnels below a large
street and finally bridges over another large street. This means there are no
large intersections where you have to wait or be cautious. In consequence the
ride is quite relaxing.</p>
<p>The only potential stop along the way is a small airfield where planes are
occasionally landing and starting. But that’s fun to watch, so I don’t mind
waiting for a minute or two.</p>
<div class="startvideo"><div class="video-container">
<div class="ytplayer" data-id="lx6VtG5S-AI"></div>
</div></div>
<script src="/public/youtube.js" type="text/javascript"></script>
<h2 id="numbers">Numbers</h2>
<p>There are about 250 work days in a year in Baden-Württemberg, excluding
weekends and public holidays. Of those I was missing for 34 days for vacations,
being sick and doing home office. On 11 days I used public transportation, on 5
days a car. That leaves 200 days on which I cycled to work.</p>
<p>The distance to work is 6 km each way, so in total I cycled 2400 km just to get
to work and back. I need 20 minutes to work and 15 minutes back, so 117 hours
were spent cycling.</p>
<p><a href="/public/cycling/T-Randonneur_Apex_2x10.jpg"><img src="/public/cycling/T-Randonneur_Apex_2x10_small.jpg" alt="T-Randonneur (mine is in green though)" /></a></p>
<p>I had two flat tires, one accident and needed one major part replacement in the
entire year. But to be fair I also cycled in my free time and it was the first
large replacement my bicycle needed since I started using it 15.000 km ago.</p>
<p><a href="/public/cycling/IMG_20170928_183309.jpg"><img src="/public/cycling/IMG_20170928_183309_small.jpg" alt="New parts" /></a></p>
<h2 id="animal-nature">Animal nature</h2>
<p>On one day in summer I counted the wild animals that I saw on my short trip to
work, which included 4 rabbits, 1 deer with fawn, 1 white stork with young in
the nest as well as a group of 21 white storks in the fields. A few weeks ago I
was happy to see an elusive black stork flying towards the closeby forest for
the first time. On another occurence there was a Heron successfully catching a
snake.</p>
<p><a href="/public/cycling/IMG_20170602_162849.jpg"><img src="/public/cycling/IMG_20170602_162849_small.jpg" alt="Storks" /></a></p>
<p>Apart from wild animals there are of course many people walking their dogs, and
the pet home on the way features lots of dogs behind the fence as well as a few
cats walking around outside.</p>
<p><a href="/public/cycling/IMG_20171016_162141.jpg"><img src="/public/cycling/IMG_20171016_162141_small.jpg" alt="Lazy cats" /></a></p>
<h2 id="winter-is-coming">Winter is coming</h2>
<p>Appropriate clothing is important since cycling exposes you to wind and
precipitation.</p>
<p>In summer a t-shirt and short pants are usually fine. For colder times
dual-function pants for office and cycling are great, otherwise jeans also
work.</p>
<p>During rains I certainly don’t want to wear jeans. Then I also need to bring
another set of clothes to switch to once I’m in the office. The most important
measure for rainy days is to have good timing, since I have flexible work times
and the rains are usually not equally strong for hours. Weather forecasts can
be remarkably inaccurate though.</p>
<p><a href="/public/cycling/IMG_20160118_144437.jpg"><img src="/public/cycling/IMG_20160118_144437_small.jpg" alt="Cycling in snow (photo from 2016)" /></a></p>
<p>I use wind-proof jackets and shoes as well as gloves and a balaclava in winter.
Snow and ice are rather rare and not so strong here, so they are not a big
issue. The above photo was taken two years ago and that represents about the
worst winter conditions I ever had to cycle in.</p>
<h2 id="conclusion">Conclusion</h2>
<p><a href="/public/cycling/IMG_20170526_183012.jpg"><img src="/public/cycling/IMG_20170526_183012_small.jpg" alt="The road ahead" /></a></p>
<p>I’ll keep cycling to work in the foreseeable future. I don’t think this post is
of much use to anyone, but I had fun reflecting my last year of cycling to
work. Thanks for reading. Comments on <a href="https://news.ycombinator.com/item?id=16420271">Hacker News</a>.</p>
Turning Down a Blockchain Job Offer2018-01-22T00:00:00+01:00https://hookrace.net/blog/turning-down-blockchain
<p>I have recently received a job offer to work on a blockchain implementation. While the offer was very generous, I had to turn it down. In this post I want to collect the thoughts that went into my decision process leading to this conclusion.</p>
<!--more-->
<h2 id="introduction">Introduction</h2>
<p>I have not touched blockchain technology much before, other than reading the original <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin paper</a> shortly after it was released and accepting Bitcoin donations for <a href="https://ddnet.org/">DDNet</a>. But I have always used those donations pretty much immediately, and thus I have no investment in cryptocurrencies either way and am thus impartial to their fate.</p>
<h2 id="the-good">The Good</h2>
<p>I consider the Bitcoin idea to be quite interesting as an electronic cash system that doesn’t require any trusted intermediary. This is a small and simple idea that combines old cryptographic ideas into something new.</p>
<p>The job I got offered had a significantly higher salary attached than my current job. The new job would have been remote, thus enabling me to spend more time with my girlfriend and live in any location I wish.</p>
<p>I would have been able to work in a nice programming language, developing a blockchain implementation from the ground up with a small group of colleagues.</p>
<h2 id="the-bad">The Bad</h2>
<p>Proof of work in the form of mining is currently the most common form of trustless consensus in blockchain systems. It requires immense amounts of computational hardware in the form of ASICs and GPUs as well as energy to supply them. The only function of this is to prevent any one actor to take over more than 50% of the mining market and thus controlling the currency. This is a large price to pay for replacing trust.</p>
<p>Looking at it the other way: Trust can be seen as a shortcut that humanity has used for a long time in order not to require expensive proof of work. For most applications this still works fine.</p>
<p>Companies are doing initial coin offerings (ICOs), giving out their own cryptotoken to public investors, circumventing a proper IPO process. While companies often claim that their tokens will have another purpose, this is usually not true.</p>
<p>Reading about blockchain is horrible because of all the hype, cargo cult and politics surrounding it. Everyone who is talking positively about cryptocurrencies and making hyperbolic claims seems to have invested in them. So they might be trying to pump up the value of cryptocurrencies in order to profit themselves. It might be that this aspect is so dominant that the actual technological details are less relevant. It feels like talking to a religious group that is trying to convert you to the one true faith that will solve all problems you and society have. I wish it was possible to filter out everyone who had money invested in the fate of cryptocurrencies in those discussions.</p>
<p>It seems like people already know that they want to use a blockchain even before they understand what the problem is, basically a solution in search of problems. The main use of cryptocurrencies right now seems to be as a speculation device.</p>
<h2 id="the-conclusion">The Conclusion</h2>
<p>In the end I guess I don’t want to be associated with the current state of the blockchain ecosystem. So even if the job offer is great, I have to deny it. It took me a long time to realize this for myself and getting my priorities straight.</p>
<p>Discussion on <a href="https://news.ycombinator.com/item?id=16205776">Hacker News</a>.</p>
HANA C++ Development Environment and Processes2017-12-08T00:00:00+01:00https://hookrace.net/blog/hana-cpp-development
<p>I started working as a C++ developer in the HANA Core Platform team at SAP in Walldorf, Germany more than a year ago. In this time I have gotten some insights into the development environment and processes. I will use this post to illustrate them by the example of adding a small new feature and explaining the steps on the way of getting it released. Some of this will be specific to HANA, some to my team inside HANA, some to my own system.</p>
<!--more-->
<p><img src="/public/office.jpg" alt="Office" /></p>
<h2 id="compilation">Compilation</h2>
<p><a href="https://www.sap.com/products/hana.html">HANA</a> is the in-house database powering many of SAP’s products. It is written in C++ with Python tests, and the entire code base lives inside of a single git repository. Hundreds of developers from all around the world are developing on about 10 million lines of C++ code and 15 million lines of Python tests.</p>
<p>Since HANA is deployed on Linux exclusively, many developers are using Linux on their workstations as well. So far Windows is still supported as a development environment, but this will change in 2019, leaving Linux as the only choice. During day-to-day work you still get to interact with Windows a bit (more than I would prefer), since the Laptop has Windows with the traditional Microsoft Office products on it. But since the actual development happens on Linux, I am happy enough, being able to use the xmonad window manager and software environment I have gotten used to over the last decade.</p>
<p>HANA is deployed on some <a href="https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/appliances.html#categories=certified&order=MemoryDesc">rather impressive machines</a>, so in order to test the code adequately the developer workstations are quite beefy as well. I am typing this text from a 20 core / 40 thread Xeon E5-2660 v3 CPU with 128 GB of RAM.</p>
<p><img src="/public/htop.png" alt="htop on developer machine" /></p>
<p>Still compiling HANA is no quick feat. Expect something around 2 hours to build it from scratch on your local machine (illustrated below), and be prepared for some heat output, which is especially noticeable in summer. About half of the colleagues have chosen to move their workstations to the data center and access it remotely instead of working on it locally. Since they are still in one of our SAP buildings in Walldorf, the latency is low enough that a direct X connection is fast enough to be nearly undistinguishable from a local application.</p>
<p><img src="/public/full-build.png" alt="Illustration of full HANA build on developer machine" /></p>
<p>So if even your beefy developer machine is not enough, how do you compile the product faster? Obviously by combining all of the developer machines together in a compile cluster:</p>
<p><img src="/public/compile-cluster.png" alt="Compile Cluster Overview" /></p>
<p>Using the compile cluster the build time gets reduced to 16 minutes and now linking takes half of that time. Right now we are still using the gold linker, but after bug fixes lld should cut down the linking time significantly.</p>
<h2 id="coding">Coding</h2>
<p>But for the code in my team we have another approach. Instead of running the full database and testing using Python tests we have extensive unit tests, literally covering our entire code. Every single line is covered and about 98% of all regions. In order to submit any new code you have to write a unit test that will cover this exact code and include it in the same change.</p>
<p>Expect 5 minutes to build and run the unit tests from scratch, while building and running them after a change can be as fast as 15 seconds if you didn’t change much and thus not much has to be recompiled.</p>
<p>People use quite a variety of IDEs and code editors, with Sublime Text, Qt Creator, Eclipse, Visual Studio, emacs and vim being popular choices (in no particular order). Personally I like to work directly on terminals, so I use vim with the <a href="https://github.com/Valloric/YouCompleteMe">YouCompleteMe</a> semantic code completion (and go to definition, type analysis and some more) based on clang/llvm. I use a <a href="https://github.com/def-/ycmd/commit/2a7124bcb5730f301e3e73a4af81316bbff81926">custom YCM version</a> which also shows me the size and alignment of a variable along with its type:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>plangen::PipelineBuilder & => hex::plangen::PipelineBuilder & (size: 128 B, align: 16 B)
</code></pre></div></div>
<p>Using vim and other command line tools gives me the advantage that I can work just fine as long as I have an SSH connection to my workstation, nothing else required.</p>
<p>Let’s start by coming up with a useful new feature, tracing the name of the <code class="language-plaintext highlighter-rouge">DummyNoRun</code> operator:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="cp">#pragma once
</span>
<span class="cp">#include</span> <span class="cpf">"hex/planex/framework/OperatorBases.hpp"</span><span class="cp">
#include</span> <span class="cpf">"hex/planex/planviz/HasReversedConnection.hpp"</span><span class="cp">
</span>
<span class="cp">#include</span> <span class="cpf">"warnings/clang_warnings_hard.h"</span><span class="cp">
#include</span> <span class="cpf">"warnings/gcc_warnings_veryhard.h"</span><span class="cp">
</span>
<span class="k">namespace</span> <span class="n">hex</span> <span class="p">{</span>
<span class="k">namespace</span> <span class="n">operators</span> <span class="p">{</span>
<span class="k">class</span> <span class="nc">DummyNoRunOp</span> <span class="k">final</span>
<span class="o">:</span> <span class="k">public</span> <span class="n">planviz</span><span class="o">::</span><span class="n">HasReversedConnection</span><span class="p">,</span>
<span class="k">public</span> <span class="n">planex</span><span class="o">::</span><span class="n">NoRunOperator</span>
<span class="p">{</span>
<span class="kt">void</span> <span class="n">traceOperatorName</span><span class="p">(</span><span class="n">ltt</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">os</span><span class="p">)</span> <span class="k">const</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}</span> <span class="c1">// namespace operators</span>
<span class="p">}</span> <span class="c1">// namespace hex</span>
<span class="cp">#include</span> <span class="cpf">"warnings/clang_warnings_end.h"</span><span class="cp">
#include</span> <span class="cpf">"warnings/gcc_warnings_end.h"</span></code></pre></figure>
<p>The implementation is quite simple as well:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="kt">void</span> <span class="n">DummyNoRunOp</span><span class="o">::</span><span class="n">traceOperatorName</span><span class="p">(</span><span class="n">ltt</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">os</span><span class="p">)</span> <span class="k">const</span>
<span class="p">{</span>
<span class="n">os</span> <span class="o"><<</span> <span class="s">"DummyNoRunOp"</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>For HANA development we use C++14 at this moment. The only unusual thing is that we don’t use the STL, but instead have our own implementation of it, the LTT, which enforces allocator use.</p>
<p>Our build system wraps around cmake, so it is quite reasonable to use. After adding the cpp file to <code class="language-plaintext highlighter-rouge">CMakeLists.txt</code> we can locally run our unit tests and verify that nothing broke so far.</p>
<p>But when we’re compiling the code with <code class="language-plaintext highlighter-rouge">hm b -b Optimized unitHexOperators</code> (read as: HappyMake build <code class="language-plaintext highlighter-rouge">Optimized</code> build [with optimizations and assertions] of the <code class="language-plaintext highlighter-rouge">unitHexOperators</code> target) we get an unexpected result:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In file included from ../../hex/plangen/PredInitOpGenerator.cpp:14:0:
../../hex/operators/table/DummyNoRunOp.hpp:16:10: error: ‘virtual void hex::operators::DummyNoRunOp::traceOperatorName(ltt::ostream&) const’ can be marked override [-Werror=suggest-override]
void traceOperatorName(ltt::ostream& os) const;
^~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
ERROR: subcommand failed
</code></pre></div></div>
<p>Our warning levels are quite high and all warnings are converted to errors. We even enable warnings in headers (and disable them again at the end in order to not affect other headers that might be included later). In this case the fix is quite simple:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"> <span class="kt">void</span> <span class="n">traceOperatorName</span><span class="p">(</span><span class="n">ltt</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">os</span><span class="p">)</span> <span class="k">const</span> <span class="k">override</span><span class="p">;</span></code></pre></figure>
<p>Now our code builds and the unit tests succeed:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ hm b -b Optimized runtests_hex
...
[==========] 728 tests from 147 test cases ran. (11617 ms total)
[ PASSED ] 728 tests.
YOU HAVE 2 DISABLED TESTS
Build started: 2017-11-29 16:26:40.459482
Build finished: 2017-11-29 16:26:52.988541
Elapsed time: 12.529s
Command count: 11 (0.9 per sec)
Log directory: /home/dXXXXXX/src2/build/Optimized/hm_log/build/24
SUCCESS
</code></pre></div></div>
<p>We can also run the code coverage based on our unit tests locally using CheckBot. Unsurprisingly we will find out that our code is not covered yet:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>###############################################################
CheckBot - Review Details
###############################################################
###############################################################
CheckBot - Summary
###############################################################
______________
General Issues
______________
[CheckHexCodeCoverageClang] (error) hex/operators/table/DummyNoRunOp.cpp: 3 of 3 lines uncovered, 0.00% coverage!
[CheckHexCodeCoverageClang] (error) Line coverage only at 99.99%
[CheckHexCodeCoverageClang] (info) Coverage report can be found at
file://///home/dXXXXXX/src2/build/linuxx86_64-clangcov-release_hex_with_code_coverage/hexCoverageClang/index.html
CheckBot detected 3 issues (2 errors, 1 info).
error ... CheckHexCodeCoverageClang (681748.81 msec)
Overall score: -2
(Legend: ok = +1, info = +1, warning = -1, error = -2)
</code></pre></div></div>
<p>At the bottom of the report we see the “Overall score: -2”, which indicates that our change would not be allowed to be merged by Gerrit. So let’s take a closer look, inside of the linked report we find our code:</p>
<p><img src="/public/llvm-cov.png" alt="llvm-cov html report" /></p>
<p>Google Test/Mock based unit tests are used to test every part of the code in our team. 100% unit test line coverage is achieved with a development version of llvm-cov. Compared to the gcc based gcov/lcov coverage tools, the LLVM based llvm-cov is in active development, runs much faster and can show you what region of a line is actually covered, allowing finer granularity.</p>
<p>So let’s write a unit test to satisfy the 100% line coverage requirement:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="cp">#include</span> <span class="cpf">"hex/operators/table/DummyNoRunOp.hpp"</span><span class="cp">
#include</span> <span class="cpf">"hex/test/HexTestBase.hpp"</span><span class="cp">
#include</span> <span class="cpf">"hex/planex/test/sandbox/SimpleOperatorSandbox.hpp"</span><span class="cp">
</span>
<span class="cp">#include</span> <span class="cpf">"warnings/clang_warnings_hard.h"</span><span class="cp">
#include</span> <span class="cpf">"warnings/gcc_warnings_hard.h"</span><span class="cp">
#include</span> <span class="cpf">"warnings/msvc_warnings_hard.h"</span><span class="cp">
</span>
<span class="k">using</span> <span class="k">namespace</span> <span class="o">::</span><span class="n">testing</span><span class="p">;</span>
<span class="k">namespace</span> <span class="n">hex</span> <span class="p">{</span>
<span class="k">namespace</span> <span class="n">operators</span> <span class="p">{</span>
<span class="k">namespace</span> <span class="n">test</span> <span class="p">{</span>
<span class="k">class</span> <span class="nc">UnitDummyNoRunOp</span> <span class="o">:</span> <span class="k">public</span> <span class="n">HexTestBase</span><span class="o"><></span>
<span class="p">{</span>
<span class="nl">protected:</span>
<span class="n">planex</span><span class="o">::</span><span class="n">SimpleOperatorSandbox</span><span class="o"><</span><span class="n">DummyNoRunOp</span><span class="o">></span> <span class="n">m_sb</span><span class="p">{</span><span class="n">alloc</span><span class="p">()};</span>
<span class="n">ltt</span><span class="o">::</span><span class="n">ostringstream</span> <span class="n">m_oss</span><span class="p">{</span><span class="n">alloc</span><span class="p">()};</span>
<span class="p">};</span>
<span class="n">TEST_F</span><span class="p">(</span><span class="n">UnitDummyNoRunOp</span><span class="p">,</span> <span class="n">traceOperatorName</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">m_oss</span> <span class="o"><<</span> <span class="n">planex</span><span class="o">::</span><span class="n">Operator</span><span class="o">::</span><span class="n">TraceName</span><span class="p">{</span><span class="n">m_sb</span><span class="p">.</span><span class="n">op</span><span class="p">()};</span>
<span class="n">EXPECT_THAT</span><span class="p">(</span><span class="n">m_oss</span><span class="p">.</span><span class="n">c_str</span><span class="p">(),</span> <span class="n">StrEq</span><span class="p">(</span><span class="s">"UnitDummyNoRunOp"</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span> <span class="c1">// namespace test</span>
<span class="p">}</span> <span class="c1">// namespace operators</span>
<span class="p">}</span> <span class="c1">// namespace hex</span></code></pre></figure>
<p>For more coarse-grained testing there are also JSON based component tests as well Python based integration tests available.</p>
<h2 id="local-verification">Local Verification</h2>
<p>We should also make use of some static code checkers, like clang-format and clang-tidy, before we push the change for remote testing. After all, this ensures a consistent style, fixes some common problems and reduces the mental load on the manual reviewers, since they can rely on the tools instead of complaining about the same trivial nitpick on every change:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>###############################################################
CheckBot - Review Details
###############################################################
_____________________________________________
hex/operators/table/test/UnitDummyNoRunOp.cpp
_____________________________________________
[CheckClangFormat] (info) This file is subject to a clang-format code style. CheckBot can reformat the file, see [...]
line 2,0 - 2,35:
#include "hex/test/HexTestBase.hpp"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[CheckClangFormat] (warning)
#include "hex/test/HexTestBase.hpp"
should be deleted
line 3,0 - 3,0:
#include "hex/planex/test/sandbox/SimpleOperatorSandbox.hpp"
^
[CheckClangFormat] (warning)
#include "hex/test/HexTestBase.hpp"
should be added
</code></pre></div></div>
<p>Luckily clang-format can even fix the problem on its own by just adding a parameter.</p>
<h2 id="remote-verification">Remote Verification</h2>
<p>Finally we can push the change to Gerrit. Now our local work is done and we can start working on something else. Meanwhile automated builds and tests will happen on Linux and Windows machines, as well as a coverage run and various static code analyzers (that we were too lazy to run locally).</p>
<p>By default just our unit tests and component tests are compiled and executed. But if you want to build the full HANA database and run the Python tests a separate profile is available for that and can be turned on in the Gerrit web interface for specific changes.</p>
<p><img src="/public/gerrit.png" alt="Gerrit Overview after successful builds" /></p>
<h2 id="code-review">Code Review</h2>
<p>Once we have verified that everything works fine on Linux with GCC and Clang as well as Windows with MSVC, we can add a reviewer to our change and set the change to “Ready for Review”. A second reviewer will be chosen automatically. If you are relatively sure that your change will survive the tests and don’t feel like waiting for them, you can of course also add the reviewers straight after submitting the change. But your reviewers might not be happy if they start reviewing already and meanwhile you are pushing new patch sets to fix compilation, tests as well as automatic warnings and suggestions.</p>
<p>For non-trivial changes further patch sets would follow to integrate the suggestions by the reviewers, until everyone is (reasonably) happy. Finally the change can be submitted and is out of our mind.</p>
<h2 id="merging">Merging</h2>
<p>Except that there is still one thing missing. Our change is now inside of our component’s <code class="language-plaintext highlighter-rouge">hex</code> branch, but not yet in the global <code class="language-plaintext highlighter-rouge">orange</code> branch. The merge from <code class="language-plaintext highlighter-rouge">hex</code> to <code class="language-plaintext highlighter-rouge">orange</code> will run a huge number of correctness and performance tests before allowing our changes in. That’s why we only run it after we have collected a few changes. We also use a staging branch <code class="language-plaintext highlighter-rouge">hex2orange</code>.</p>
<ul>
<li>Initially we merge the <code class="language-plaintext highlighter-rouge">hex</code> branch into <code class="language-plaintext highlighter-rouge">hex2orange</code> and try to merge <code class="language-plaintext highlighter-rouge">hex2orange</code> into <code class="language-plaintext highlighter-rouge">orange</code>.</li>
<li>Meanwhile everyone can keep developing new features as well as bug fixes on the <code class="language-plaintext highlighter-rouge">hex</code> branch.</li>
<li>If the merge from <code class="language-plaintext highlighter-rouge">hex2orange</code> to <code class="language-plaintext highlighter-rouge">orange</code> fails, a fix for the issue will be submitted to <code class="language-plaintext highlighter-rouge">hex</code> and can finally be cherry-picked to <code class="language-plaintext highlighter-rouge">hex2orange</code>.</li>
</ul>
<p>But we don’t merge <code class="language-plaintext highlighter-rouge">hex</code> into <code class="language-plaintext highlighter-rouge">hex2orange</code> again, until we landed in the <code class="language-plaintext highlighter-rouge">orange</code> branch. Otherwise the new features from <code class="language-plaintext highlighter-rouge">hex</code> could cause new test failures while we were fixing the old ones. In the worst case we would never reach <code class="language-plaintext highlighter-rouge">orange</code>.</p>
<p>Once the colleagues give a green light on all tests, the merge goes in and we can sit and wait for bugs for our new feature to roll in.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope you found something interesting in this post. On a related note, SAP HANA is <a href="https://jobs.sap.com/search/?q=SAPhanacareers&locationsearch=&utm_source=DennisFelsing">hiring right now in a few locations</a>.</p>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=15890028">Hacker News</a> and <a href="https://www.reddit.com/r/programming/comments/7if7fa/hana_c_development_environment_and_processes/">r/programming</a>.</p>
C++ Quiz #12017-12-01T00:00:00+01:00https://hookrace.net/blog/cpp-quiz-1
<p>What is the output of this small snippet that is based on real C++ code?</p>
<!--more-->
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="cp">#include</span> <span class="cpf"><iostream></span><span class="cp">
</span>
<span class="k">struct</span> <span class="nc">Foo</span>
<span class="p">{</span>
<span class="n">Foo</span><span class="p">()</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"standard"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="n">Foo</span><span class="p">(</span><span class="k">const</span> <span class="n">Foo</span><span class="o">&</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"copy"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="n">Foo</span><span class="p">(</span><span class="kt">int</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"int"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="n">Foo</span><span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"int, int"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="n">Foo</span><span class="p">(</span><span class="k">const</span> <span class="n">Foo</span><span class="o">&</span><span class="p">,</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"Foo, int"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="n">Foo</span><span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="k">const</span> <span class="n">Foo</span><span class="o">&</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"int, Foo"</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="p">};</span>
<span class="kt">void</span> <span class="nf">f</span><span class="p">(</span><span class="n">Foo</span><span class="p">)</span> <span class="p">{}</span>
<span class="k">struct</span> <span class="nc">Bar</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">m_i</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">m_j</span><span class="p">;</span>
<span class="n">Bar</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">f</span><span class="p">(</span><span class="n">Foo</span><span class="p">(</span><span class="n">m_i</span><span class="p">,</span> <span class="n">m_j</span><span class="p">));</span>
<span class="n">f</span><span class="p">(</span><span class="n">Foo</span><span class="p">(</span><span class="n">m_i</span><span class="p">));</span>
<span class="n">Foo</span><span class="p">(</span><span class="n">m_i</span><span class="p">,</span> <span class="n">m_j</span><span class="p">);</span>
<span class="n">Foo</span><span class="p">(</span><span class="n">m_i</span><span class="p">);</span>
<span class="n">Foo</span><span class="p">(</span><span class="n">m_i</span><span class="p">,</span> <span class="n">m_j</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">Bar</span><span class="p">();</span>
<span class="p">}</span></code></pre></figure>
<p>The code compiles without warnings with clang++ 5.0.0 as well as g++ 7.2.0 using <code class="language-plaintext highlighter-rouge">-Wall -Wextra</code>.</p>
<script language="javascript">
function toggle() {
var spoiler = document.getElementById("spoiler");
spoiler.style.display = (spoiler.style.display == "block") ? "none" : "block";
}
</script>
<h2><a href="javascript:toggle();">Solution</a></h2>
<div id="spoiler" style="display: none">
<p>The preliminaries are simple:</p>
<ul>
<li>We have a struct Foo with a few different constructors, each of which prints a text that represents its parameter types.</li>
<li>We have a struct Bar with two members, <code class="language-plaintext highlighter-rouge">m_i</code> and <code class="language-plaintext highlighter-rouge">m_j</code>.</li>
<li>The <code class="language-plaintext highlighter-rouge">main</code> function calls the standard constructor of Bar.</li>
</ul>
<p>The constructor of Bar is where the magic happens, so let’s go through it line by line:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">f(Foo(m_i, m_j))</code> creates a temporary Foo object by calling the Foo constructor with <code class="language-plaintext highlighter-rouge">m_i</code> and <code class="language-plaintext highlighter-rouge">m_j</code> as parameters, so <code class="language-plaintext highlighter-rouge">int, int</code> is printed. The resulting Foo object is then passed to the function <code class="language-plaintext highlighter-rouge">f</code>.</li>
<li><code class="language-plaintext highlighter-rouge">f(Foo(m_i))</code> works analogously, it calls the Foo constructor with <code class="language-plaintext highlighter-rouge">m_i</code> as parameter, so <code class="language-plaintext highlighter-rouge">int</code> is printed. The resulting Foo object is then passed to the function <code class="language-plaintext highlighter-rouge">f</code>.</li>
<li><code class="language-plaintext highlighter-rouge">Foo(m_i, m_j)</code> works the same as the first line, except it doesn’t pass the resulting temporary Foo object to a function, so it is destroyed again immediately, <code class="language-plaintext highlighter-rouge">int, int</code> is printed again.</li>
<li>So far so good, but now look closely. <code class="language-plaintext highlighter-rouge">Foo(m_i)</code> surprisingly behaves entirely differently from all the previous lines. It does <em>not</em> call the Foo constructor with <code class="language-plaintext highlighter-rouge">m_i</code> as the parameter. Instead it creates a Foo object of the name <code class="language-plaintext highlighter-rouge">m_i</code>, just like <code class="language-plaintext highlighter-rouge">Foo m_i</code> would. So <code class="language-plaintext highlighter-rouge">standard</code> is printed.</li>
<li>Now the last line looks just like the third line, but it still does something different. Why? Because the name <code class="language-plaintext highlighter-rouge">m_i</code> is no longer referring to the <code class="language-plaintext highlighter-rouge">int m_i</code> member of Bar. Instead <code class="language-plaintext highlighter-rouge">m_i</code> is now referring to a local variable of type <code class="language-plaintext highlighter-rouge">Foo</code>, so <code class="language-plaintext highlighter-rouge">Foo(m_i, m_j)</code> prints <code class="language-plaintext highlighter-rouge">Foo, int</code>.</li>
</ol>
<h2 id="explanation">Explanation</h2>
<p>Thus spoke the C++ standard:</p>
<blockquote>
<p>§8.3 Meaning of declarators</p>
<p>[…]</p>
<p>6 In a declaration <code class="language-plaintext highlighter-rouge">T D</code> where <code class="language-plaintext highlighter-rouge">D</code> has the form</p>
<p><em>( D1 )</em></p>
<p>the type of the contained <em>declarator-id</em> is the same as that of the contained <em>declarator-id</em> in the declaration</p>
<p><code class="language-plaintext highlighter-rouge">T D1</code></p>
<p>Parentheses do not alter the type of the embedded <code class="language-plaintext highlighter-rouge">declarator-id</code>, but they can alter the binding of complex declarators.</p>
</blockquote>
<p>This kind of ambiguous parsing can lead to dangerous situations, as is documented in <a href="https://wiki.sei.cmu.edu/confluence/display/cplusplus/DCL53-CPP.+Do+not+write+syntactically+ambiguous+declarations">DCL53-CPP</a>:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="cp">#include</span> <span class="cpf"><mutex></span><span class="cp">
</span>
<span class="k">static</span> <span class="n">std</span><span class="o">::</span><span class="n">mutex</span> <span class="n">m</span><span class="p">;</span>
<span class="k">static</span> <span class="kt">int</span> <span class="n">shared_resource</span><span class="p">;</span>
<span class="kt">void</span> <span class="nf">increment_by_42</span><span class="p">()</span> <span class="p">{</span>
<span class="n">std</span><span class="o">::</span><span class="n">unique_lock</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">mutex</span><span class="o">></span><span class="p">(</span><span class="n">m</span><span class="p">);</span>
<span class="n">shared_resource</span> <span class="o">+=</span> <span class="mi">42</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>The code looks like it locks the mutex <code class="language-plaintext highlighter-rouge">m</code> while modifiying <code class="language-plaintext highlighter-rouge">shared_resource</code>, but instead a new mutex <code class="language-plaintext highlighter-rouge">m</code> is created, shadowing the global mutex <code class="language-plaintext highlighter-rouge">m</code>.</p>
<h2 id="future-directions">Future Directions</h2>
<p>The upcoming Clang version will have a warning for this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>y.cpp:25:12: warning: parentheses were disambiguated as redundant parentheses around declaration of variable named
'm_i' [-Wvexing-parse]
Foo(m_i);
^~~~~
y.cpp:25:9: note: add enclosing parentheses to perform a function-style cast
Foo(m_i);
^
( )
y.cpp:25:12: note: remove parentheses to silence this warning
Foo(m_i);
^ ~
</code></pre></div> </div>
</div>
time.gif2017-08-11T00:00:00+02:00https://hookrace.net/blog/time.gif
<p>This is an endless GIF that always shows the current time in UTC:</p>
<!--more-->
<p><img src="/time.gif" alt="time.gif (reload if it fails)" /></p>
<p><a href="https://github.com/def-/time.gif">Source Code</a></p>
<p>(From reports it doesn’t seem to work on Safari, other browsers should be fine.)</p>
<p>time.gif is written in Haskell and works by dynamically generating each frame of the GIF and slowly feeding them over the HTTP connection.</p>
<p>There is no guarantee that this GIF shows a reasonable time and this is just for fun anyway, so better don’t build your next project based on the time from this GIF. If my server is overloaded, you can try compiling it yourself and run it locally.</p>
<p>Update: Optimized, runs at less than 1% CPU with 500 simultaneous connections open, LZW encoding reduces bandwidth from 4 KB/s to 300 B/s.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="c1">-- Wait for incoming connections and start delivering a GIF to them</span>
<span class="n">loop</span> <span class="o">::</span> <span class="kt">Int</span> <span class="o">-></span> <span class="kt">FrameSignal</span> <span class="o">-></span> <span class="kt">Socket</span> <span class="o">-></span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="n">loop</span> <span class="n">delay</span> <span class="n">frameSignal</span> <span class="n">sock</span> <span class="o">=</span> <span class="kr">do</span>
<span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="kr">_</span><span class="p">)</span> <span class="o"><-</span> <span class="n">accept</span> <span class="n">sock</span>
<span class="n">forkIO</span> <span class="o">$</span> <span class="n">body</span> <span class="n">conn</span>
<span class="n">loop</span> <span class="n">delay</span> <span class="n">frameSignal</span> <span class="n">sock</span>
<span class="kr">where</span>
<span class="n">body</span> <span class="n">c</span> <span class="o">=</span> <span class="kr">do</span>
<span class="n">f</span> <span class="o"><-</span> <span class="n">receiveMSignal</span> <span class="n">frameSignal</span>
<span class="n">sendAll</span> <span class="n">c</span> <span class="o">$</span> <span class="n">msg</span> <span class="o">$</span> <span class="n">initialFrame</span> <span class="p">(</span><span class="n">delay</span> <span class="p">`</span><span class="n">div</span><span class="p">`</span> <span class="mi">10000</span><span class="p">)</span> <span class="n">f</span>
<span class="n">nextFrame</span> <span class="n">c</span>
<span class="n">nextFrame</span> <span class="n">c</span> <span class="o">=</span> <span class="kr">do</span>
<span class="n">f</span> <span class="o"><-</span> <span class="n">receiveMSignal</span> <span class="n">frameSignal</span>
<span class="n">sendAll</span> <span class="n">c</span> <span class="o">$</span> <span class="n">frame</span> <span class="p">(</span><span class="n">delay</span> <span class="p">`</span><span class="n">div</span><span class="p">`</span> <span class="mi">10000</span><span class="p">)</span> <span class="n">f</span>
<span class="n">nextFrame</span> <span class="n">c</span>
<span class="n">msg</span> <span class="n">content</span> <span class="o">=</span> <span class="kt">B</span><span class="o">.</span><span class="n">intercalate</span> <span class="s">"</span><span class="se">\r\n</span><span class="s">"</span>
<span class="p">[</span> <span class="s">"HTTP/1.0 200 OK"</span>
<span class="p">,</span> <span class="s">"Server: gifstream/0.1"</span>
<span class="p">,</span> <span class="s">"Content-Type: image/gif"</span>
<span class="p">,</span> <span class="s">"Content-Transfer-Encoding: binary"</span>
<span class="p">,</span> <span class="s">"Cache-Control: no-cache"</span>
<span class="p">,</span> <span class="s">"Cache-Control: no-store"</span>
<span class="p">,</span> <span class="s">"Cache-Control: no-transform"</span>
<span class="p">,</span> <span class="s">""</span>
<span class="p">,</span> <span class="n">content</span>
<span class="p">]</span></code></pre></figure>
<p>This is cannibalized from <a href="https://github.com/def-/gifstream">gifstream</a>, which lets you play snake and have people watch a GIF livestream. It was actually created as a Christmas exercise for students of the Programming Paradigms course at KIT.</p>
<p><img src="https://raw.githubusercontent.com/def-/gifstream/master/snake.gif" alt="snake" /></p>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=33358486">Hacker News (2022-10-27)</a> (<a href="https://news.ycombinator.com/item?id=14996715">previous thread from 2017-08-12</a>)</p>
Nim Code Coverage2016-11-01T00:00:00+01:00https://hookrace.net/blog/nim-code-coverage
<p>Creating code coverage reports with Nim is surprisingly easy. You can simply
use the good old gcov and lcov tools. Nim can be told to insert its own line
information with the <code class="language-plaintext highlighter-rouge">--debugger:native</code> command line parameter.</p>
<p>Here’s the small example program we’re looking at:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">var</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"foo"</span>
<span class="n">echo</span> <span class="s">"bar"</span></code></pre></figure>
<!--more-->
<p>Note that if we change the condition to <code class="language-plaintext highlighter-rouge">if false:</code> the Nim compiler optimizes
the impossible code away and it will not count as uncovered. The same thing can
happen with entire uncovered functions when optimizations and dead code
elimination are enabled.</p>
<p>The file is saved as <code class="language-plaintext highlighter-rouge">x.nim</code>. Here’s the script I use to create the code
coverage report:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/sh</span>
<span class="nb">rm</span> <span class="nt">-rf</span> <span class="k">*</span>.info html nimcache
nim <span class="nt">--debugger</span>:native <span class="nt">--passC</span>:--coverage <span class="nt">--passL</span>:--coverage c x
lcov <span class="nt">--base-directory</span> <span class="nb">.</span> <span class="nt">--directory</span> <span class="nb">.</span> <span class="nt">--zerocounters</span> <span class="nt">-q</span>
./x
lcov <span class="nt">--base-directory</span> <span class="nb">.</span> <span class="nt">--directory</span> <span class="nb">.</span> <span class="nt">-c</span> <span class="nt">-o</span> x.info
lcov <span class="nt">--remove</span> x.info <span class="s2">"lib/*"</span> <span class="nt">-o</span> x.info <span class="c"># remove Nim system libs from coverage</span>
genhtml <span class="nt">-o</span> html x.info</code></pre></figure>
<p>You can look at the <a href="/public/coverage-minimal/">final html report</a> generated.</p>
A Taste of Haskell2016-10-21T00:00:00+02:00https://hookrace.net/blog/a-taste-of-haskell
<p>In this post I want to highlight a few fun aspects of the Haskell programming language. The purpose is to give you a taste of Haskell so that you will want to learn more of it. Don’t consider this as a tutorial or guide but rather as a starting point, as it is based on a short talk I held at work, which in turn is based on my favorite material from holding practical courses about Haskell at university.</p>
<p>Let’s start by seeing how programmers compare Haskell to a mainstream programming language, for example Java:</p>
<!--more-->
<p><a href="http://hammerprinciple.com/therighttool"><img src="/public/haskell/hammer1.png" alt="Java vs Haskell" /></a>
Image Source: <a href="http://hammerprinciple.com/therighttool">Hammer Principle</a></p>
<p>Doesn’t sound that good for Java, what do people say about Haskell?</p>
<p><a href="http://hammerprinciple.com/therighttool"><img src="/public/haskell/hammer2.png" alt="Haskell vs Java" /></a>
Image Source: <a href="http://hammerprinciple.com/therighttool">Hammer Principle</a></p>
<p>Sounds much better and makes it seem like a reasonable endeavour to learn Haskell! So what is Haskell actually?</p>
<blockquote>
<p>Haskell is a purely functional, lazily evaluated, statically typed programming language with type inference.</p>
</blockquote>
<p>While this definition sounds complicated you should understand more of it after reading this post. Let’s start with the basics: What’s the difference between an imperative programming language and a functional one? In imperative programming the basic operation is changing a stored value. In functional programming the basic operation is applying a function to arguments.</p>
<p><a href="https://haskell-lang.org/">Haskell</a> itself was designed in 1990 by a scientific committee with the purpose of being a basis for functional programming research. The <a href="https://www.haskell.org/ghc/">Glasgow Haskell Compiler</a> (GHC) is the most popular implementation and the one we will be using. You can get it as part of the <a href="https://www.haskell.org/platform/">Haskell Platform</a> as well. Haskell code can be compiled (<code class="language-plaintext highlighter-rouge">ghc</code>), but also interpreted (<code class="language-plaintext highlighter-rouge">ghci</code>, interactive).</p>
<p>In the rest of this blog post we will look at some key features of Haskell interactively, so you can get your own installation of GHC and follow along and experiment with the code.</p>
<h2 id="arithmetic">Arithmetic</h2>
<p>I belive that the best way to learn a programming language is by playing around with it. So that’s what we’re going to do now. I’ll show a few examples and explain the cool Haskell features that we encounter on the way.</p>
<p>We start out by running ghci, the interactive Glasgow Haskell Compiler interpreter (indicated by <code class="language-plaintext highlighter-rouge">λ></code>). We also create a single file <code class="language-plaintext highlighter-rouge">tut.hs</code> which we will use to write some more advanced Haskell code and load in the interpreter.</p>
<p>For starters let’s do some basic arithmetic in GHCi, replacing our dusty old calculator:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="mi">2</span><span class="o">+</span><span class="mi">3</span>
<span class="mi">5</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">4</span> <span class="o">*</span> <span class="mi">5</span>
<span class="mi">20</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">5</span> <span class="o">/</span> <span class="mi">6</span>
<span class="mf">0.8333333333333334</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">2</span> <span class="o">^</span> <span class="mi">10</span>
<span class="mi">1024</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">2</span> <span class="o">^</span> <span class="mi">1000</span>
<span class="mi">10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376</span></code></pre></figure>
<p>We see that Integers can be of unbounded size, instead of the usual limits of 32 or 64 bits in many programming languages. Of course if you really want them there are also more efficient machine ints in Haskell:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="p">(</span><span class="mi">2</span> <span class="o">::</span> <span class="kt">Int</span><span class="p">)</span> <span class="o">^</span> <span class="mi">10</span>
<span class="mi">1024</span>
<span class="err">λ</span><span class="o">></span> <span class="p">(</span><span class="mi">2</span> <span class="o">::</span> <span class="kt">Int</span><span class="p">)</span> <span class="o">^</span> <span class="mi">100</span>
<span class="mi">0</span></code></pre></figure>
<p>Using the <code class="language-plaintext highlighter-rouge">:: Int</code> notation we specifiy that the number <code class="language-plaintext highlighter-rouge">2</code> is explicitly of type <code class="language-plaintext highlighter-rouge">Int</code> instead of being automatically inferred to be of type <code class="language-plaintext highlighter-rouge">Integer</code>. While we’re looking at types, there are also floating point numbers of course:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="mi">2</span> <span class="o">**</span> <span class="mi">10</span>
<span class="mf">1024.0</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">2</span> <span class="o">**</span> <span class="mi">1000</span>
<span class="mf">1.0715086071862673e301</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">2</span> <span class="o">**</span> <span class="mi">10000</span>
<span class="kt">Infinity</span></code></pre></figure>
<p>As usual, floating point numbers are of limited precision, so at some point we just reach “approximately infinity”. Let’s do some more math:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">pi</span>
<span class="mf">3.141592653589793</span>
<span class="err">λ</span><span class="o">></span> <span class="n">sin</span>
<span class="o"><</span><span class="n">interactive</span><span class="o">>:</span><span class="mi">12</span><span class="o">:</span><span class="mi">1</span><span class="o">:</span>
<span class="kt">No</span> <span class="kr">instance</span> <span class="n">for</span> <span class="p">(</span><span class="kt">Show</span> <span class="p">(</span><span class="n">a0</span> <span class="o">-></span> <span class="n">a0</span><span class="p">))</span>
<span class="p">(</span><span class="n">maybe</span> <span class="n">you</span> <span class="n">haven't</span> <span class="n">applied</span> <span class="n">enough</span> <span class="n">arguments</span> <span class="n">to</span> <span class="n">a</span> <span class="n">function</span><span class="o">?</span><span class="p">)</span>
<span class="n">arising</span> <span class="n">from</span> <span class="n">a</span> <span class="n">use</span> <span class="kr">of</span> <span class="err">‘</span><span class="n">print</span><span class="err">’</span>
<span class="kt">In</span> <span class="n">the</span> <span class="n">first</span> <span class="n">argument</span> <span class="kr">of</span> <span class="err">‘</span><span class="n">print</span><span class="err">’</span><span class="p">,</span> <span class="n">namely</span> <span class="err">‘</span><span class="n">it</span><span class="err">’</span>
<span class="kt">In</span> <span class="n">a</span> <span class="n">stmt</span> <span class="kr">of</span> <span class="n">an</span> <span class="n">interactive</span> <span class="kt">GHCi</span> <span class="n">command</span><span class="o">:</span> <span class="n">print</span> <span class="n">it</span></code></pre></figure>
<p>Looks like <code class="language-plaintext highlighter-rouge">sin</code> is a function, but the error message is already confusing. Looking at the part in brackets gives us a good hint: <em>maybe you haven’t applied enough arguments to a function?</em>. Let’s check out the type of <code class="language-plaintext highlighter-rouge">sin</code>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="kr">type</span> <span class="n">sin</span>
<span class="n">sin</span> <span class="o">::</span> <span class="kt">Floating</span> <span class="n">a</span> <span class="o">=></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span></code></pre></figure>
<p>Invoking <code class="language-plaintext highlighter-rouge">:type</code> tells us that <code class="language-plaintext highlighter-rouge">sin</code> is a function that takes a value of type <code class="language-plaintext highlighter-rouge">a</code> and returns a value of type <code class="language-plaintext highlighter-rouge">a</code>, where <code class="language-plaintext highlighter-rouge">a</code> is a floating point number. So let’s pass a number to <code class="language-plaintext highlighter-rouge">sin</code>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">sin</span> <span class="mf">0.5</span>
<span class="mf">0.479425538604203</span>
<span class="err">λ</span><span class="o">></span> <span class="n">sin</span> <span class="mi">10</span>
<span class="o">-</span><span class="mf">0.5440211108893698</span>
<span class="err">λ</span><span class="o">></span> <span class="n">div</span> <span class="mi">15</span> <span class="mi">6</span>
<span class="mi">2</span>
<span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">div</span>
<span class="n">div</span> <span class="o">::</span> <span class="kt">Integral</span> <span class="n">a</span> <span class="o">=></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">div</code> is another function. As you can see it accepts two parameters, both of type <code class="language-plaintext highlighter-rouge">a</code> and finally returns a value of type <code class="language-plaintext highlighter-rouge">a</code>. In this case <code class="language-plaintext highlighter-rouge">a</code> has to be an <code class="language-plaintext highlighter-rouge">Integral</code>, some kind of integer-like type. We can even ask GHCi what an <code class="language-plaintext highlighter-rouge">Integral</code> is supposed to be:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">info</span> <span class="kt">Integral</span>
<span class="kr">class</span> <span class="p">(</span><span class="kt">Real</span> <span class="n">a</span><span class="p">,</span> <span class="kt">Enum</span> <span class="n">a</span><span class="p">)</span> <span class="o">=></span> <span class="kt">Integral</span> <span class="n">a</span> <span class="kr">where</span>
<span class="n">quot</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span>
<span class="n">rem</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span>
<span class="n">div</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span>
<span class="n">mod</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span>
<span class="n">quotRem</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">a</span><span class="p">)</span>
<span class="n">divMod</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">a</span><span class="p">)</span>
<span class="n">toInteger</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">Integer</span>
<span class="c1">-- Defined in ‘GHC.Real’</span>
<span class="kr">instance</span> <span class="kt">Integral</span> <span class="kt">Word</span> <span class="c1">-- Defined in ‘GHC.Real’</span>
<span class="kr">instance</span> <span class="kt">Integral</span> <span class="kt">Integer</span> <span class="c1">-- Defined in ‘GHC.Real’</span>
<span class="kr">instance</span> <span class="kt">Integral</span> <span class="kt">Int</span> <span class="c1">-- Defined in ‘GHC.Real’</span></code></pre></figure>
<p>Integral is a type class which requires a few functions to be defined for the type, like <code class="language-plaintext highlighter-rouge">quot</code> and <code class="language-plaintext highlighter-rouge">rem</code>. We also see that an Integral type needs to have all properties of a <code class="language-plaintext highlighter-rouge">Real</code> and an <code class="language-plaintext highlighter-rouge">Enum</code> type. Three types are known to GHCi which adhere to this type class: <code class="language-plaintext highlighter-rouge">Word</code>, <code class="language-plaintext highlighter-rouge">Integer</code> and <code class="language-plaintext highlighter-rouge">Int</code></p>
<p>It looks a bit awkward to write <code class="language-plaintext highlighter-rouge">div 15 6</code>, so Haskell offers some syntactic sugar to use the <code class="language-plaintext highlighter-rouge">div</code> function in infix notation:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="mi">15</span> <span class="p">`</span><span class="n">div</span><span class="p">`</span> <span class="mi">6</span>
<span class="mi">2</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">15</span> <span class="p">`</span><span class="n">mod</span><span class="p">`</span> <span class="mi">6</span>
<span class="mi">3</span></code></pre></figure>
<p>The same thing works in reverse to use operators as regular functions, using brackets:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">2</span> <span class="mi">3</span>
<span class="mi">5</span>
<span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span>
<span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="o">::</span> <span class="kt">Num</span> <span class="n">a</span> <span class="o">=></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span></code></pre></figure>
<p>Of course there are quite a lot of other functions that we won’t have time to explore, but you can always explore them yourself in the <a href="http://hackage.haskell.org/package/base-4.9.0.0/docs/Prelude.html">documentation</a> or using <a href="https://www.haskell.org/hoogle/">Hoogle</a>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">max</span> <span class="mi">2</span> <span class="mi">3</span>
<span class="mi">3</span>
<span class="err">λ</span><span class="o">></span> <span class="n">max</span> <span class="mi">5</span> <span class="mi">3</span>
<span class="mi">5</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">2</span> <span class="o">></span> <span class="mi">3</span>
<span class="kt">False</span>
<span class="err">λ</span><span class="o">></span> <span class="n">not</span> <span class="p">(</span><span class="mi">2</span> <span class="o">></span> <span class="mi">3</span><span class="p">)</span>
<span class="kt">True</span>
<span class="err">λ</span><span class="o">></span> <span class="kt">True</span> <span class="o">&&</span> <span class="kt">False</span>
<span class="kt">False</span>
<span class="err">λ</span><span class="o">></span> <span class="kt">True</span> <span class="o">&&</span> <span class="kt">True</span>
<span class="kt">True</span></code></pre></figure>
<h2 id="lists">Lists</h2>
<p>Lists are the most important data structure for us, they are simply a collection of values of the same type:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">info</span> <span class="kt">[]</span>
<span class="kr">data</span> <span class="kt">[]</span> <span class="n">a</span> <span class="o">=</span> <span class="kt">[]</span> <span class="o">|</span> <span class="n">a</span> <span class="o">:</span> <span class="p">[</span><span class="n">a</span><span class="p">]</span> <span class="c1">-- Defined in ‘GHC.Types’</span>
<span class="o">...</span></code></pre></figure>
<p>What this definition tells us is that a list is a data type that is either an empty list (<code class="language-plaintext highlighter-rouge">[]</code>) or a value concatenated with a list itself (<code class="language-plaintext highlighter-rouge">a : [a]</code>). So the data type is recursively defined, referring to itself. With this knowledge we can create basic lists ourselves:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="kt">[]</span>
<span class="kt">[]</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">1</span><span class="o">:</span><span class="kt">[]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="mi">1</span><span class="o">:</span><span class="mi">2</span><span class="o">:</span><span class="kt">[]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">]</span></code></pre></figure>
<p>We can also use some syntactic sugar to create lists instead:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span></code></pre></figure>
<p>We just said that a list is supposed to be a collection of values of the same type. What happens if we try to break that?</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="kt">True</span><span class="p">]</span>
<span class="o"><</span><span class="n">interactive</span><span class="o">>:</span><span class="mi">27</span><span class="o">:</span><span class="mi">2</span><span class="o">:</span>
<span class="kt">No</span> <span class="kr">instance</span> <span class="n">for</span> <span class="p">(</span><span class="kt">Num</span> <span class="kt">Bool</span><span class="p">)</span> <span class="n">arising</span> <span class="n">from</span> <span class="n">the</span> <span class="n">literal</span> <span class="err">‘</span><span class="mi">1</span><span class="err">’</span>
<span class="kt">In</span> <span class="n">the</span> <span class="n">expression</span><span class="o">:</span> <span class="mi">1</span>
<span class="kt">In</span> <span class="n">the</span> <span class="n">expression</span><span class="o">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="o">....</span><span class="p">]</span>
<span class="kt">In</span> <span class="n">an</span> <span class="n">equation</span> <span class="n">for</span> <span class="err">‘</span><span class="n">it</span><span class="err">’</span><span class="o">:</span> <span class="n">it</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="o">....</span><span class="p">]</span></code></pre></figure>
<p>Good, that shouldn’t work and indeed it doesn’t. The error message tells us that the list is assumed to be a list of booleans because of the final value. But <code class="language-plaintext highlighter-rouge">1</code> is not a boolean value, so there is no valid type for this list.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="o">..</span><span class="mi">20</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">9</span><span class="p">,</span><span class="mi">11</span><span class="p">,</span><span class="mi">13</span><span class="p">,</span><span class="mi">15</span><span class="p">,</span><span class="mi">17</span><span class="p">,</span><span class="mi">19</span><span class="p">]</span></code></pre></figure>
<p>We can store values in variables, but the name “variable” might confuse you a bit, because variables can not be overwritten:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">xs</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">xs</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">6</span><span class="p">]</span></code></pre></figure>
<p>The seconds line creates a new variable <code class="language-plaintext highlighter-rouge">xs</code> that makes the old one invisible in the new scope. Actually variables are always immutable in Haskell. That means you can easily share access to the same data because there is no way in which it can be overwritten:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">ys</span> <span class="o">=</span> <span class="n">xs</span></code></pre></figure>
<p>Let’s look at a few common list operations in Haskell:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">head</span> <span class="n">xs</span>
<span class="mi">1</span>
<span class="err">λ</span><span class="o">></span> <span class="n">tail</span> <span class="n">xs</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">init</span> <span class="n">xs</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">last</span> <span class="n">xs</span>
<span class="mi">6</span>
<span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">2</span> <span class="n">xs</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">length</span> <span class="n">xs</span>
<span class="mi">6</span></code></pre></figure>
<p><a href="http://learnyouahaskell.com/starting-out#an-intro-to-lists"><img src="/public/haskell/listmonster.png" alt="List Visualization" /></a>
Image Source: <a href="http://learnyouahaskell.com/starting-out#an-intro-to-lists">Learn You a Haskell for Great Good!</a></p>
<p>Knowing how lists are implemented we can easily implement our own definitions of these functions:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="c1">-- data [] a = [] | a : [a]</span>
<span class="n">head'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">x</span></code></pre></figure>
<p>This definition uses pattern matching. The pattern is <code class="language-plaintext highlighter-rouge">(x:xs)</code> where <code class="language-plaintext highlighter-rouge">x</code> and <code class="language-plaintext highlighter-rouge">xs</code> are variables delimited by the cons (<code class="language-plaintext highlighter-rouge">:</code>). We use the knowledge of the definition to split up the passed value into two parts, the first element <code class="language-plaintext highlighter-rouge">x</code> and the rest of the list <code class="language-plaintext highlighter-rouge">xs</code>. Then the result of our function is simply the first element <code class="language-plaintext highlighter-rouge">x</code>.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">l</span> <span class="n">tut</span>
<span class="err">λ</span><span class="o">></span> <span class="n">head'</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="mi">1</span>
<span class="err">λ</span><span class="o">></span> <span class="n">head'</span> <span class="kt">[]</span>
<span class="o">***</span> <span class="kt">Exception</span><span class="o">:</span> <span class="n">tut</span><span class="o">.</span><span class="n">hs</span><span class="o">:</span><span class="mi">1</span><span class="o">:</span><span class="mi">1</span><span class="o">-</span><span class="mi">16</span><span class="o">:</span> <span class="kt">Non</span><span class="o">-</span><span class="n">exhaustive</span> <span class="n">patterns</span> <span class="kr">in</span> <span class="n">function</span> <span class="n">head'</span></code></pre></figure>
<p>Oops, we didn’t cover one possible pattern, the empty list! We can simply add a definition for that to throw a nicer error message:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">head'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="n">error</span> <span class="s">"head of empty list undefined"</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">r</span>
<span class="p">[</span><span class="mi">1</span> <span class="kr">of</span> <span class="mi">1</span><span class="p">]</span> <span class="kt">Compiling</span> <span class="kt">Main</span> <span class="p">(</span> <span class="n">tut</span><span class="o">.</span><span class="n">hs</span><span class="p">,</span> <span class="n">interpreted</span> <span class="p">)</span>
<span class="kt">Ok</span><span class="p">,</span> <span class="n">modules</span> <span class="n">loaded</span><span class="o">:</span> <span class="kt">Main</span><span class="o">.</span>
<span class="err">λ</span><span class="o">></span> <span class="n">head'</span> <span class="kt">[]</span>
<span class="o">***</span> <span class="kt">Exception</span><span class="o">:</span> <span class="n">head</span> <span class="kr">of</span> <span class="n">empty</span> <span class="n">list</span> <span class="n">undefined</span></code></pre></figure>
<p>Unfortunately there is nothing usefuly we can return for <code class="language-plaintext highlighter-rouge">head []</code>, because it’s impossible for us to construct a value of any arbitrary type. Let’s look at the type of <code class="language-plaintext highlighter-rouge">head'</code>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">head'</span>
<span class="n">head'</span> <span class="o">::</span> <span class="p">[</span><span class="n">t</span><span class="p">]</span> <span class="o">-></span> <span class="n">t</span></code></pre></figure>
<p>The type was automatically inferred for us, but we can also write it down explicitly if we want to make sure that we don’t break it in the future:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">head'</span> <span class="o">::</span> <span class="p">[</span><span class="n">a</span><span class="p">]</span> <span class="o">-></span> <span class="n">a</span></code></pre></figure>
<p>This means that <code class="language-plaintext highlighter-rouge">head'</code> is a function that takes a list of values of type <code class="language-plaintext highlighter-rouge">a</code> as its parameter and returns a single value of type <code class="language-plaintext highlighter-rouge">a</code>.</p>
<p>If we had defined head’ for integer lists only, we could return a <code class="language-plaintext highlighter-rouge">0</code> for example as the default value:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">head'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">head'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">x</span></code></pre></figure>
<p>But if we now check the inferred type we notice that it changed, now only lists containing numbers are supported:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">head'</span>
<span class="n">head'</span> <span class="o">::</span> <span class="kt">Num</span> <span class="n">a</span> <span class="o">=></span> <span class="p">[</span><span class="n">a</span><span class="p">]</span> <span class="o">-></span> <span class="n">a</span></code></pre></figure>
<p>So in this case there is nothing better we can do than throw an error.</p>
<p>Analogously we can define the <code class="language-plaintext highlighter-rouge">tail</code> function:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">tail'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="kt">[]</span>
<span class="n">tail'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">xs</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">tail'</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">tail'</span> <span class="kt">[]</span>
<span class="kt">[]</span></code></pre></figure>
<p>This was easy enough. How can we define the <code class="language-plaintext highlighter-rouge">last</code> function?</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">last</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="mi">5</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">last'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="n">error</span> <span class="s">"last of empty list undefined"</span>
<span class="n">last'</span> <span class="p">[</span><span class="n">x</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
<span class="n">last'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">last'</span> <span class="n">xs</span></code></pre></figure>
<p>Here the idea is to reduce the problem to something that we can solve. If the list only contains a single element, we know that this exact element is the last one. If the list has more than one element, we remove the first element of the list and recursively call <code class="language-plaintext highlighter-rouge">last'</code> on the rest of the list. Yet again the last element of an empty list makes no sense, so we throw an error.</p>
<p>What about defining init? The list of all values but the last?</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">init</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">init'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="n">error</span> <span class="s">"init of empty list undefined"</span>
<span class="n">init'</span> <span class="p">[</span><span class="n">x</span><span class="p">]</span> <span class="o">=</span> <span class="kt">[]</span>
<span class="n">init'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">x</span> <span class="o">:</span> <span class="n">init'</span> <span class="n">xs</span></code></pre></figure>
<p>The idea here is a bit more complicated. We know that <code class="language-plaintext highlighter-rouge">init</code> of a list with one element <code class="language-plaintext highlighter-rouge">[x]</code> is the empty list <code class="language-plaintext highlighter-rouge">[]</code>. If the list has more than one element we know that <code class="language-plaintext highlighter-rouge">init</code> contains the first element <code class="language-plaintext highlighter-rouge">x</code>, concatenated with <code class="language-plaintext highlighter-rouge">init</code> of the rest of the list.</p>
<p>Finally let’s define the length function, which works similarly:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">length'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">length'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="n">length'</span> <span class="n">xs</span></code></pre></figure>
<p>The length of an empty list is 0. The length of a list with more than 0 elements is 1 plus the length of the rest of the list. These functions read like mathematical definitions of the properties they are encoding.</p>
<h2 id="higher-order-functions">Higher-order Functions</h2>
<p>So we have defined our first basic functions and noticed that there is no magic happening in Haskell’s standard library. Instead all of these functions are easily implementable. Now let’s look at some more advanced operations that we can perform on lists, for example mapping a function to a list, thus applying it to each value in the list:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">f</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">x</span>
<span class="err">λ</span><span class="o">></span> <span class="n">map</span> <span class="n">f</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span></code></pre></figure>
<p>It’s a common theme in functional programming to pass functions as parameters to higher order functions. But it’s a bit annoying to define a named function explicitly all the time, so instead we can quickly create an unnamed function, called a lambda function, instead:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">map</span> <span class="p">(</span><span class="nf">\</span><span class="n">x</span> <span class="o">-></span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span></code></pre></figure>
<p>You can see the lambda function <code class="language-plaintext highlighter-rouge">(\x -> x * 2)</code>, which is specified to take a parameter <code class="language-plaintext highlighter-rouge">x</code> and return <code class="language-plaintext highlighter-rouge">x * 2</code>. Of course Haskell has some sweet syntactic sugar to do this even more succinctly by just writing <code class="language-plaintext highlighter-rouge">(*2)</code>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">map</span> <span class="p">(</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span></code></pre></figure>
<p>Let’s see how we can define our own <code class="language-plaintext highlighter-rouge">map</code> function:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">map'</span> <span class="n">f</span> <span class="kt">[]</span> <span class="o">=</span> <span class="kt">[]</span></code></pre></figure>
<p>The base case is easy: When we get an empty list passed, the result is also an empty list.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">map'</span> <span class="n">f</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">f</span> <span class="n">x</span> <span class="o">:</span> <span class="n">map'</span> <span class="n">f</span> <span class="n">xs</span></code></pre></figure>
<p>Otherwise we take the first element <code class="language-plaintext highlighter-rouge">x</code> in the list, apply the function <code class="language-plaintext highlighter-rouge">f</code> to it and create a new list with this new value as the initial element. We recurse and apply map to the rest of the list in the same way.</p>
<p>At this point we can take a look at the actual implementations in <a href="http://hackage.haskell.org/package/base-4.9.0.0/docs/src/GHC.Base.html#map">GHC’s standard library</a> and we notice that <code class="language-plaintext highlighter-rouge">map</code> is implemented exactly as we wrote it.</p>
<p>Note that we’re not modifying the passed data structure directly, instead we create a new one. Actually in Haskell there is no way to modify data structures, they are all immutable. And functions are pure, so there is no way for a function to have any side effects other than returning a value directly. That’s also why we have referential transparency: When you call a function with the same inputs, it will always return the same output.</p>
<p>Let’s turn to the next function, <code class="language-plaintext highlighter-rouge">filter</code>, which keeps only those elements in a list which fulfil a predicate:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">filter</span> <span class="n">odd</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span></code></pre></figure>
<p>Filter is also easy to implement:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">filter'</span> <span class="n">f</span> <span class="kt">[]</span> <span class="o">=</span> <span class="kt">[]</span>
<span class="n">filter'</span> <span class="n">f</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span>
<span class="o">|</span> <span class="n">f</span> <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">:</span> <span class="n">filter'</span> <span class="n">f</span> <span class="n">xs</span>
<span class="o">|</span> <span class="n">otherwise</span> <span class="o">=</span> <span class="n">filter'</span> <span class="n">f</span> <span class="n">xs</span></code></pre></figure>
<p>The guarded equation at the start of the line means that we check a boolean. When <code class="language-plaintext highlighter-rouge">f x</code> is true we return the first line, including <code class="language-plaintext highlighter-rouge">x</code>, otherwise we don’t include <code class="language-plaintext highlighter-rouge">x</code> in the rest of the list. Finally we recurse with the rest of the list, until we reach the base case for the empty list.</p>
<p>By implementing a few functions on list a common pattern emerges. Let’s see how to implement the sum over a list:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">sum</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="mi">15</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">sum'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">sum'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">sum'</span> <span class="n">xs</span></code></pre></figure>
<p>We already implemented the length of a list:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">length'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">length'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="n">length'</span> <span class="n">xs</span></code></pre></figure>
<p>How do we define the <code class="language-plaintext highlighter-rouge">and</code> function over an entire list?</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">and</span> <span class="p">[</span><span class="kt">True</span><span class="p">,</span> <span class="kt">False</span><span class="p">,</span> <span class="kt">True</span><span class="p">]</span>
<span class="kt">False</span>
<span class="err">λ</span><span class="o">></span> <span class="n">and</span> <span class="p">[</span><span class="kt">True</span><span class="p">,</span> <span class="kt">True</span><span class="p">,</span> <span class="kt">True</span><span class="p">]</span>
<span class="kt">True</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">and'</span> <span class="kt">[]</span> <span class="o">=</span> <span class="kt">True</span>
<span class="n">and'</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">x</span> <span class="o">&&</span> <span class="n">and'</span> <span class="n">xs</span></code></pre></figure>
<p>Notice the similarity by now?</p>
<p>The functions <code class="language-plaintext highlighter-rouge">sum</code>, <code class="language-plaintext highlighter-rouge">length</code> and <code class="language-plaintext highlighter-rouge">and</code> all are built in pretty much the same way. So we can create an abstraction over this pattern, which is called a right-associative fold, or <code class="language-plaintext highlighter-rouge">foldr</code> in short:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">foldr'</span> <span class="n">f</span> <span class="n">i</span> <span class="kt">[]</span> <span class="o">=</span> <span class="n">i</span>
<span class="n">foldr'</span> <span class="n">f</span> <span class="n">i</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">x</span> <span class="p">`</span><span class="n">f</span><span class="p">`</span> <span class="n">foldr'</span> <span class="n">f</span> <span class="n">i</span> <span class="n">xs</span></code></pre></figure>
<p>Using this pattern it becomes trivial to define these functions:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">sum''</span> <span class="n">xs</span> <span class="o">=</span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span> <span class="n">xs</span></code></pre></figure>
<p>Even better, we don’t need to write down the last parameter, <code class="language-plaintext highlighter-rouge">xs</code>:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">sum''</span> <span class="o">=</span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span></code></pre></figure>
<p>What’s going on there? Actually in Haskell when you pass a parameter to a function, a new function is returned. So you can consider each function as accepting a single parameter, then returning a new function, which is applied to the next parameter, and so on.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="mi">15</span>
<span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">foldr'</span>
<span class="n">foldr'</span> <span class="o">::</span> <span class="p">(</span><span class="n">t</span> <span class="o">-></span> <span class="n">t1</span> <span class="o">-></span> <span class="n">t1</span><span class="p">)</span> <span class="o">-></span> <span class="n">t1</span> <span class="o">-></span> <span class="p">[</span><span class="n">t</span><span class="p">]</span> <span class="o">-></span> <span class="n">t1</span>
<span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span>
<span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="o">::</span> <span class="kt">Num</span> <span class="n">t1</span> <span class="o">=></span> <span class="n">t1</span> <span class="o">-></span> <span class="p">[</span><span class="n">t1</span><span class="p">]</span> <span class="o">-></span> <span class="n">t1</span>
<span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span>
<span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span> <span class="o">::</span> <span class="kt">Num</span> <span class="n">t1</span> <span class="o">=></span> <span class="p">[</span><span class="n">t1</span><span class="p">]</span> <span class="o">-></span> <span class="n">t1</span>
<span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="n">foldr'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="mi">0</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span> <span class="o">::</span> <span class="p">(</span><span class="kt">Enum</span> <span class="n">t1</span><span class="p">,</span> <span class="kt">Num</span> <span class="n">t1</span><span class="p">)</span> <span class="o">=></span> <span class="n">t1</span></code></pre></figure>
<p>So in our definition of <code class="language-plaintext highlighter-rouge">sum''</code> we don’t need to have a parameter and put it into <code class="language-plaintext highlighter-rouge">foldr'</code>, instead we can also have no parameter and just return the function that is returned from <code class="language-plaintext highlighter-rouge">(foldr' (+) 0)</code>, which takes a list as its parameter.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">and''</span> <span class="o">=</span> <span class="n">foldr'</span> <span class="p">(</span><span class="o">&&</span><span class="p">)</span> <span class="kt">True</span>
<span class="n">length''</span> <span class="o">=</span> <span class="n">foldr'</span> <span class="p">(</span><span class="nf">\</span><span class="n">x</span> <span class="n">y</span> <span class="o">-></span> <span class="mi">1</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span> <span class="mi">0</span></code></pre></figure>
<p>We can even define <code class="language-plaintext highlighter-rouge">map</code> and filter with foldr:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">map''</span> <span class="n">f</span> <span class="o">=</span> <span class="n">foldr'</span> <span class="p">(</span><span class="nf">\</span><span class="n">x</span> <span class="n">xs</span> <span class="o">-></span> <span class="n">f</span> <span class="n">x</span> <span class="o">:</span> <span class="n">xs</span><span class="p">)</span> <span class="kt">[]</span>
<span class="n">filter''</span> <span class="n">f</span> <span class="o">=</span> <span class="n">foldr'</span> <span class="n">go</span> <span class="kt">[]</span>
<span class="kr">where</span> <span class="n">go</span> <span class="n">x</span> <span class="n">xs</span>
<span class="o">|</span> <span class="n">f</span> <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">:</span> <span class="n">xs</span>
<span class="o">|</span> <span class="n">otherwise</span> <span class="o">=</span> <span class="n">xs</span></code></pre></figure>
<h2 id="lazy-evaluation-and-sharing">Lazy Evaluation and Sharing</h2>
<p>While working with lists in Haskell you might wonder what happens if we never reach the base case? We have lazy evaluation in Haskell, which tells us that a data structure is only evaluated when it’s actually needed. So we have no problem handling lists of infinite size:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">x</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">5</span> <span class="n">x</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">5</span> <span class="o">$</span> <span class="n">map</span> <span class="p">(</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> <span class="n">x</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">5</span> <span class="o">$</span> <span class="n">map</span> <span class="p">(</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> <span class="o">$</span> <span class="n">filter</span> <span class="n">odd</span> <span class="n">x</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">14</span><span class="p">,</span><span class="mi">18</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">x</span>
<span class="o">...</span></code></pre></figure>
<p>We can even combine multiple infinite lists without any problem:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">y</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="o">..</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="kr">let</span> <span class="n">z</span> <span class="o">=</span> <span class="n">zipWith</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="n">x</span> <span class="n">y</span>
<span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">10</span> <span class="n">z</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">11</span><span class="p">,</span><span class="mi">14</span><span class="p">,</span><span class="mi">17</span><span class="p">,</span><span class="mi">20</span><span class="p">,</span><span class="mi">23</span><span class="p">,</span><span class="mi">26</span><span class="p">,</span><span class="mi">29</span><span class="p">]</span></code></pre></figure>
<p>Of course <code class="language-plaintext highlighter-rouge">length</code> won’t work as it will never reach the base case:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">length</span> <span class="n">z</span>
<span class="o">^</span><span class="kt">CInterrupted</span><span class="o">.</span></code></pre></figure>
<p>Let’s go a step back and create an infinite list of ones:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">ones</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="o">..</span><span class="p">]</span></code></pre></figure>
<p>The problem is that this is very inefficient, because we recalculate new elements all the time and have to store them in memory.</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">length</span> <span class="n">ones</span></code></pre></figure>
<p><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/intro1-1.svg" alt="ones1" /><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/intro1-2.svg" alt="ones2" /><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/intro1-3.svg" alt="ones3" /></p>
<p>You can easily see how the memory usage grows. A more efficient solution is to use recursion in the definition:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">ones'</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">:</span> <span class="n">ones'</span></code></pre></figure>
<p>We can use <code class="language-plaintext highlighter-rouge">ones'</code> itself in the definition of <code class="language-plaintext highlighter-rouge">ones'</code>, just referring to it. Thanks to lazy evaluation the next value is only evaluated when it is needed, so when we call:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">3</span> <span class="n">ones'</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span></code></pre></figure>
<p>We see that <code class="language-plaintext highlighter-rouge">1</code> is the first value of the list, we recurse into <code class="language-plaintext highlighter-rouge">ones'</code>, we see that <code class="language-plaintext highlighter-rouge">1</code> is the next value we get, we recurse into <code class="language-plaintext highlighter-rouge">ones'</code> and finally we get the last <code class="language-plaintext highlighter-rouge">1</code>.</p>
<p><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/intro2.svg" alt="ones1" /><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/intro2.svg" alt="ones2" /><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/intro2.svg" alt="ones3" /></p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">length</span> <span class="n">ones'</span></code></pre></figure>
<p>Of course there is no useful result, it’s an infinite calculation, but at least we don’t create a list of infinite size, instead referring to ourselves.</p>
<p>Let’s look at another fun list combinator that takes two lists and creates a new one out of them:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="o">:</span><span class="n">t</span> <span class="n">zipWith</span>
<span class="n">zipWith</span> <span class="o">::</span> <span class="p">(</span><span class="n">a</span> <span class="o">-></span> <span class="n">b</span> <span class="o">-></span> <span class="n">c</span><span class="p">)</span> <span class="o">-></span> <span class="p">[</span><span class="n">a</span><span class="p">]</span> <span class="o">-></span> <span class="p">[</span><span class="n">b</span><span class="p">]</span> <span class="o">-></span> <span class="p">[</span><span class="n">c</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="n">zipWith</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">10</span><span class="p">]</span> <span class="p">[</span><span class="mi">100</span><span class="o">..</span><span class="mi">110</span><span class="p">]</span>
<span class="p">[</span><span class="mi">101</span><span class="p">,</span><span class="mi">103</span><span class="p">,</span><span class="mi">105</span><span class="p">,</span><span class="mi">107</span><span class="p">,</span><span class="mi">109</span><span class="p">,</span><span class="mi">111</span><span class="p">,</span><span class="mi">113</span><span class="p">,</span><span class="mi">115</span><span class="p">,</span><span class="mi">117</span><span class="p">,</span><span class="mi">119</span><span class="p">]</span></code></pre></figure>
<p>So, <code class="language-plaintext highlighter-rouge">zipWith</code> is a function that takes a function <code class="language-plaintext highlighter-rouge">(a -> b -> c)</code> as its first parameter. The second parameter is a list of values of type <code class="language-plaintext highlighter-rouge">a</code>, the third parameter is a list of values of type <code class="language-plaintext highlighter-rouge">b</code> and finally a list of type <code class="language-plaintext highlighter-rouge">c</code> is returned.</p>
<p>Of course to understand it we best implement it:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">zipWith'</span> <span class="n">f</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="p">(</span><span class="n">y</span><span class="o">:</span><span class="n">ys</span><span class="p">)</span> <span class="o">=</span> <span class="n">f</span> <span class="n">x</span> <span class="n">y</span> <span class="o">:</span> <span class="n">zipWith'</span> <span class="n">f</span> <span class="n">xs</span> <span class="n">ys</span>
<span class="n">zipWith'</span> <span class="n">f</span> <span class="n">xs</span> <span class="n">ys</span> <span class="o">=</span> <span class="kt">[]</span></code></pre></figure>
<p>We can use <code class="language-plaintext highlighter-rouge">zipWith</code> to trivially define a list of all fibonacci numbers:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">fibs</span> <span class="o">=</span> <span class="mi">0</span> <span class="o">:</span> <span class="mi">1</span> <span class="o">:</span> <span class="n">zipWith'</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="n">fibs</span> <span class="p">(</span><span class="n">tail</span> <span class="n">fibs</span><span class="p">)</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">take</span> <span class="mi">10</span> <span class="n">fibs</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">13</span><span class="p">,</span><span class="mi">21</span><span class="p">,</span><span class="mi">34</span><span class="p">]</span></code></pre></figure>
<p>The start makes sense, a list of fibs starts with 0 and 1, then the rest is defined with <code class="language-plaintext highlighter-rouge">zipWith</code> over <code class="language-plaintext highlighter-rouge">fibs</code> itself and <code class="language-plaintext highlighter-rouge">(tail fibs)</code>. This smells like magic, how can it possibly work? Functions are pure in Haskell. That means that a function always returns the same value when you call it with the same arguments. There is no way to have a variable in which you store some state. Since data structures and functions in Haskell are immutable and pure we can not change them in any way, so we can reuse them without having to recalculate them. So in this case we can refer to <code class="language-plaintext highlighter-rouge">fibs'</code> multiple times even inside the definition of <code class="language-plaintext highlighter-rouge">fibs'</code> itself.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fibs 0 : 1 : 1 : 2 : 3 : 5 : 8
tail fibs 1 : 1 : 2 : 3 : 5 : 8 :
zipWith (+) 1 : 2 : 3 : 5 : 8 : :
</code></pre></div></div>
<p><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/fib1.svg" alt="Fib1" /><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/fib2.svg" alt="Fib2" /><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/fib3.svg" alt="Fib3" /></p>
<h2 id="list-comprehensions">List Comprehensions</h2>
<p>Instead of all these <code class="language-plaintext highlighter-rouge">filter</code>s and <code class="language-plaintext highlighter-rouge">map</code>s we can also use list comprehensions:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="err">λ</span><span class="o">></span> <span class="n">map</span> <span class="p">(</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> <span class="o">$</span> <span class="n">filter</span> <span class="n">odd</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span>
<span class="err">λ</span><span class="o">></span> <span class="p">[</span><span class="n">x</span><span class="o">*</span><span class="mi">2</span> <span class="o">|</span> <span class="n">x</span> <span class="o"><-</span> <span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">],</span> <span class="n">odd</span> <span class="n">x</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span></code></pre></figure>
<p>We can even implement the sieve of eratosthenes to calculate all prime numbers as a oneliner with a list comprehension:</p>
<figure class="highlight"><pre><code class="language-haskell" data-lang="haskell"><span class="n">sieve</span> <span class="p">(</span><span class="n">p</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="o">=</span> <span class="n">p</span> <span class="o">:</span> <span class="n">sieve</span> <span class="p">[</span><span class="n">x</span> <span class="o">|</span> <span class="n">x</span> <span class="o"><-</span> <span class="n">xs</span><span class="p">,</span> <span class="n">x</span> <span class="p">`</span><span class="n">mod</span><span class="p">`</span> <span class="n">p</span> <span class="o">></span> <span class="mi">0</span><span class="p">]</span>
<span class="n">primes</span> <span class="o">=</span> <span class="n">sieve</span> <span class="p">[</span><span class="mi">2</span><span class="o">..</span><span class="p">]</span></code></pre></figure>
<p>Because in Haskell functions are pure and data is immutable we can perform equational reasoning. We can guarantee that code can be replaced by its definition. Basically this means that you can do refactoring without any risks. You know that nothing can go wrong because there is no state. Every function only takes its arguemnts and returns a value based on those, always the same value. So you can reorganize your functions however you want, the order does not matter and there is actually no order semantically enforced in Haskell.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> primes
= sieve [2..]
= 2 : sieve [x | x <- [3,4,5,6,7,8,9,10,11..], x `mod` 2 > 0]
= 2 : sieve [3,5,7,9,11..]
= 2 : 3 : sieve [x | x <- [5,7,9,11..], x `mod` 3 > 0]
= 2 : 3 : sieve [5,7,11..]
= 2 : 3 : 5 : sieve [...]
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>I hope you enjoyed this small excursion into functional programming land with Haskell. Maybe you learned something that gives you something to think when programming in your favorite programming language. If you’re interested in learning more Haskell, here are a few books you can read:</p>
<p><a href="http://www.cs.nott.ac.uk/~pszgmh/pih.html"><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/pih.jpg" alt="Programming in Haskell" /></a><a href="http://learnyouahaskell.com/"><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/haskell-book-cover.png" alt="Learn You a Haskell for Great Good!" /></a><a href="http://book.realworldhaskell.org/"><img style="display: inline; width: 33%; padding: 0;" src="/public/haskell/rwh_cover.jpg" alt="Real World Haskell" /></a></p>
<p>Discussions on <a href="https://news.ycombinator.com/item?id=12782888">Hacker News</a> and <a href="https://www.reddit.com/r/programming/comments/593ud7/a_taste_of_haskell/">r/Programming</a>.</p>
New Hardware and Hacks2016-07-03T00:00:00+02:00https://hookrace.net/blog/new-hardware-hacks
<p>In my <a href="/blog/broken-hardware-fixes-hacks-8-years/">latest post</a> I showed some
examples of how I ran mostly the same PC hardware over a period of 8 years.
Today I finally finished setting up my new PC hardware in my new home, so I can
report about what I did differently, my thought process, and some problems I
encountered and hacks I did to solve them.</p>
<p>I’m not sure if this article is interesting for anyone, but I had some fun
setting the new system up and felt like writing about it, so here we go:</p>
<!--more-->
<h2 id="case">Case</h2>
<p>I wanted a smaller case since the old one was a huge ATX tower with way too
many useless ports and fans. I still want a pretty performant system, so an
Intel NUC is out of the question. I don’t need a GPU, since the most
challenging graphics I need is DDNet client, which easily runs at 400 fps at
1920x1080 and 130 fps at 3840x2160 on Intel’s HD Graphics 530 IGP integrated on
Skylake CPUs. But it’s still nice to have a PCIe port in case you ever need it.
So Mini-ITX it is.</p>
<p>Unfortunately the <a href="http://www.streacom.com/products/db4-fanless-chassis/">Streacom DB4</a> is not available yet, as it would have made for a beautiful silent Mini-ITX cube. Instead I went with the <a href="https://www.inwin-style.com/en/gaming-chassis/Chopin">In Win Chopin</a>, which is even smaller, features no GPU port and has a small integrated PSU rated at 150 W. I was a bit worried about the PSU being strong enough, but you can read more about that later in this post. There really aren’t that many good-looking and reasonable Mini-ITX cases.</p>
<p><img class="halfimg" src="/public/hardware2/streacom-db4.jpg" alt="Streacom DB4" /><img class="halfimg" src="/public/hardware2/in-win-chopin.jpg" alt="In Win Chopin" /></p>
<p>The idea of the In Win Chopin is to use the CPU fan as the only fan to cool the
entire case, pulling air directly from the side, pushing it out of the top.
That’s a stark contrast to my old machine with a total of 9 fans whirring
inside the case.</p>
<h2 id="cpu-cooler">CPU Cooler</h2>
<p>Of course that means I need a good CPU cooler for the small space. With just 43 mm space for the cooler my choice fell on the <a href="http://noctua.at/en/nh-l9i">Noctua NH-L9i</a>. Unfortunately the manufacturer <a href="http://noctua.at/en/nh_l9i_tdp_guidelines">claims a 91 W Skylake CPU barely runs with the NH-L9i</a>. So let’s check out the CPU and measure some actual temperatures. Even though I switched to a small form factor, another goal of mine is still to keep the noise down to a level where I don’t notice it.</p>
<p><img class="halfimg" src="/public/hardware2/noctua-nh-l9i.jpg" alt="Noctua NH L9i" /><img class="halfimg" src="/public/hardware2/intel-i7-6700k.jpg" alt="Intel i7 6700k" /></p>
<h2 id="cpu">CPU</h2>
<p>There are just two 91 W Intel Skylake CPUs and my choice fell on the i7-6700k.
Since I never had a failed CPU before and planned to play around a bit with the
CPU anyway, thus voiding the warranty, I decided to get a pre-owned one for
cheap. Apparently the CPU was unused, but I wouldn’t believe that.</p>
<p>There are much nicer Xeon CPUs, especially if you’re willing to risk the
problems of running an engineering sample, but I wouldn’t be able to cool them
properly, wouldn’t need all that performance in the end and miss out on the
IGP (integrated graphics processor), needing a separate GPU instead, taking up
more space.</p>
<p>Unfortunately it turns out that Skylake can only decode 8 bit H.264 and H.265 in
hardware, while most sources are switching to 10 bit, including UltraHD
Blu-rays and Netflix. If I had the chance I might have waited for Kabylake,
Intel’s next CPU release.</p>
<p>At first I installed the i7-6700k with the regular Noctua thermal paste and
used it like that for a few days. Unfortunately the system ran louder than I wanted. And changing the fan settings was not a solution since the CPU reached 80°C at load. Clearly I needed a better solution. I already expected this and had some <a href="http://www.thermal-grizzly.com/en/products/26-conductonaut-en">Thermal Grizzly Conductonaut</a> liquid metal thermal paste ready to use.</p>
<p><img src="/public/hardware2/conductonaut.jpg" alt="Thermal Grizzly Conductonaut" /></p>
<p>Liquid metal thermal pastes conduct heat much better than regular thermal
pastes, but unfortunately they also conduct electricity, so you have to be
careful when applying them. Usually people use this thermal paste to achieve
better overclocking, but my goal is opposite, for now I want a more quiet
system running at low power. Maybe I’ll overclock if I ever need the extra
performance.</p>
<p>So I delidded my new CPU by removing the heat spreader with a razor
blade. You need a very careful hand for that as a single scratch into the PCB
could ruin your CPU. Of course this also voids the warranty if you have one. I
removed the regular thermal paste between the CPU die and the heat spreader and
applied the Conductonaut instead. The heat spreader can be reattached with
regular silicone from a hardware store. Finally I used the same Conductonaut
thermal paste between the heat spreader and the cooler as well.</p>
<p>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/JkGASegVRiM" frameborder="0" allowfullscreen=""></iframe>
</div>
</p>
<p>Finally I dared to start my system again, luckily everything still seemed to
work and I couldn’t quite believe the new temperatures: Instead of 80°C at load
I now had just 60°C. I was hoping for a 10-15°C improvement, so that was a nice
surprise.</p>
<p>My last step was to undervolt the CPU and IGP as far as they would go. You have
to be careful about system stability with this. I set up a few different work
loads to test stability. This further reduced the load temperature to 55°C.
That’s a temperature I can definitely live with and doesn’t even require full
fan speed to keep up. At regular light usage the CPU stays at 30-35°C and I
can’t even hear the fan. I don’t dare to turn off the fan entirely, since it’s
the only active cooling solution for all the other components on the mainboard
as well.</p>
<h2 id="screen">Screen</h2>
<p>In a few months <a href="http://www.bbc.co.uk/mediacentre/latestnews/2016/planet-earth-two">Planet Earth 2</a> will air in Ultra HD. Since I’m a huge fan of David Attenborough and nature documentaries in general I clearly need a screen capable of showing it. I also want to use my screen as a television at the same time, so this fits in well.</p>
<p>So the plan was to get a 40” screen with a 3840x2160 resolution, which enables me to work as if I have four 1920x1080 screens, positioned in a 2x2 grid. Luckily this is pretty simple to setup with <a href="http://xmonad.org/">xmonad</a>, my window manager. Using the <a href="http://xmonad.org/xmonad-docs/xmonad-contrib/XMonad-Layout-LayoutScreens.html">LayoutScreens module</a> I can switch between pretending to have a single large screen or 4 small ones. Since xmonad configuration is just Haskell programming more layouts would of course be possible, but so far I’m happy with this setup.</p>
<p>It turns out that there is still a problem here which I can’t quite pinpoint
yet: When I use xmonad, mpv, NetWM support, 3840x2160 resolution, and the Intel
IGP driver’s TearFree setting, memory leaks and after a few minutes of watching
some video all 32 GB of RAM are used up and a process is killed. Changing any
single component in this list gets rid of the problem. For now I use DRI3
instead of TearFree, which has slight tearing only in very special cases that I
never encounter in regular usage.</p>
<p>But one problem I imagine to have with a large screen is that the left and
right edges are so far away that they look distorted. After all when I use two
regular screens I rotate them a bit so I can look at them head-on. So instead I
wanted a curved screen to correct the distortion and get a similar effect to
having the screens rotated. I’m not really convinced of curved screens for a
pure TV setup, but as a computer screen they are great.</p>
<p>Luckily the <a href="http://www.samsung.com/de/consumer/tv-av/tv/uhd/UE40JU6740UXZG">Samsung UE40JU6740</a> fit all my requirements and was available for
cheap since it’s last year’s model. Getting it to display 3840x2160@60Hz with
reasonable colors is a bit challenging though. There are even guides about how
to <a href="https://hardforum.com/threads/2015-samsung-4k-tv-as-a-monitor-set-up-guide.1869675/">properly set these TVs up as computer
screens</a>.
In the end I have a far better picture than with my old screens, but the input
delay of 40 ms is noticeable when playing <a href="https://ddnet.org/">DDNet</a>. Since I
don’t play much I don’t mind and in case I ever need it it’s possible to get it
down to a more reasonable 20 ms at the loss of color accuracy.</p>
<p><img src="/public/hardware2/Samsung-UE40JU6740.jpg" alt="Samsung UE40JU6740" /></p>
<h2 id="mainboard">Mainboard</h2>
<p>But now I faced a new challenge: TVs generally don’t support DisplayPort, so in
order to get 3840x2160@60Hz HDMI 2.0 is required. That’s another thing Skylake
CPUs don’t support, so I had two choices: Get an external converter from DP 1.2
to HDMI 2.0, which Club 3D offers. Or alternatively get the only Mini-ITX mainboard with HDMI 2.0, the <a href="http://www.asrock.com/mb/Intel/Fatal1ty%20Z170%20Gaming-ITXac/">ASRock Z170 Gaming-ITX/ac</a>. So now I have a gaming mainboard even though I don’t want to game, oh well.</p>
<p><img class="halfimg" src="/public/hardware2/club3d.jpg" alt="Club3D adapter" /><img class="halfimg" src="/public/hardware2/asrock-z170-gaming-itx.jpg" alt="ASRock Z170 Gaming-ITX/ac" /></p>
<p>In the end both the Club 3D adapter as well as the ASRock motherboard have
nearly the same chips inside, a <a href="http://www.megachips.us/products/MCDP28_Products.php">MegaChips
MCDP28</a>.</p>
<p>Unfortunately the board still has a few issues even after having been released
nearly a year ago:</p>
<p>Using the HDMI 2.0 port at 3840x2160@60Hz causes a frame with artifacts every
few minutes, both in Windows and Linux. In Linux sometimes the screen goes
black entirely when I switch on stereo HDMI audio, while mono works. My
assumption is that this is a bandwidth or synchronization problem. So for now I
disabled HDMI audio and this seems to fix the image artifacts. I never planned
to use the HDMI port for audio since I still have my fancy amplifier and
speakers, but it would be nice to get this working.</p>
<p>Another temporary workaround to get working HDMI 2.0 audio on Linux is to user
another CRTC manually by running <code class="language-plaintext highlighter-rouge">xrandr --output DP2 --crtc 2</code>. But in my
experience it’s not 100% reliable and I’m not even sure why it works.</p>
<p>Actually ASRock <a href="http://asrock.pc.cdn.bitgravity.com/TSD/TheGuildofsupporting4kx2k@60Hz.pdf">offers a firmware
update</a>
for the MegaChips HDMI 2.0 chip, but it doesn’t run on my system and I’m still
talking to the support about it.</p>
<p>The next issue was that resuming from S3 sleep was broken, also both in Windows
and Linux. I assume this might also have to do with the HDMI 2.0 chip, but I’m
not sure yet. So far my only solution is to keep using an older BIOS version
(1.8) which does not exhibit this problem.</p>
<p>Instead I get a very strange issue after resuming from sleep. In DDNet client I
get 40 fps instead of 500 fps after the first sleep cycle. My first idea was
that it had something to do with the IGP and power saving, but I couldn’t find
a way to pinpoint it. Finally I noticed that some screens in DDNet have far
fewer FPS than others. Finally it dawned on me: The more text is shown on
screen, the slower it gets.</p>
<p>Nevertheless it had nothing to do with font rendering. Instead the font
rendering system in DDNet client, which is inherited directly from Teeworlds,
needs to call the <code class="language-plaintext highlighter-rouge">gettimeofday</code> syscall A LOT. Checking <code class="language-plaintext highlighter-rouge">dmesg</code> then confirmed
that after a S3 sleep the Linux kernel clock source switched from TSC to HPET:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Jun 28 22:49:23 al kernel: TSC synchronization [CPU#0 -> CPU#1]:
Jun 28 22:49:23 al kernel: Measured 3072272461 cycles TSC warp between CPUs, turning off TSC clock.
Jun 28 22:49:23 al kernel: tsc: Marking TSC unstable due to check_tsc_sync_source failed
Jun 28 01:17:48 al kernel: clocksource: Switched to clocksource hpet
</code></pre></div></div>
<p>Manually setting <code class="language-plaintext highlighter-rouge">tsc=reliable</code> makes the clock run backwards after a sleep
cycle. From what I read it sounds like the BIOS is erroneously changing some
TSC registers at suspend or resume. I didn’t find a way to fix these registers
again in the Linux kernel.</p>
<p>Fortunately the support quickly gave me a BIOS that is supposed to fix this
issue. Unfortunately I can’t test this since every BIOS version newer than 1.8
doesn’t resume from S3 sleep at all for me.</p>
<p>Instead I did the best I could and fixed DDNet client to call <code class="language-plaintext highlighter-rouge">clock_gettime</code>
two times per frame instead of potentially thousands of times per frame (once
for each text glyph that is rendered). Some measurement showed that getting the
time with TSC requires 18 ns, while HPET needs 4700 ns on my system, quite a
remarkable difference.</p>
<h2 id="storage">Storage</h2>
<p>As a nice bonus the mainboard also has an M.2 port on the backside, so I can run a fast <a href="http://www.samsung.com/semiconductor/products/flash-storage/client-ssd/MZHPV256HDGL?ia=831">Samsung SM951</a> SSD. Unfortunately being located on the backside of the mainboard means that air circulation is rather bad, so the SSD runs at a temperature of about 50°C.</p>
<p><img src="/public/hardware2/samsung-sm951.jpg" alt="Samsung SM951" /></p>
<p>If this becomes a problem later I can try a thin thermal pad to get contact
between the SSD and the metal plate inside the case, thus using it as a cooler.</p>
<h2 id="ram">RAM</h2>
<p>For memory I actually just bought the cheapest 32 GB DDR4 kit from Crucial at
2133 MHz. Turns out that you can undervolt the RAM a bit and still overclock it
to 2700 MHz. That helps the IGP a bit, but is otherwise not really noticeable.
It turned out that you actually need dual channel RAM to achieve
3840x2160@60Hz, so memory bandwidth is actually a concern.</p>
<p><img src="/public/hardware2/crucial-32gb.jpg" alt="Crucial 32 GB" /></p>
<h2 id="keyboard">Keyboard</h2>
<p>My choices regarding a keyboard were very limited. I wanted a flat keyboard but no rubberdome since they kept annoying me with their inconsistent feeling and bad quality. The only keyboard to fit these requirements was the <a href="http://www.cherry.de/cid/keyboards_CHERRY_MX-Board_30.htm?rdeLocaleAttr=en&cpssessionid=SID-FE6BB73E-EA37D60F&WT.mc_id=">Cherry MX-Board 3.0</a>. With MX-Blue switches it’s quite loud to type on, but makes my other keyboards feel incredibly cheap.</p>
<p><img src="/public/hardware2/cherry-mx-board-3.0.jpg" alt="Cherry MX-Board 3.0" /></p>
<h2 id="mouse">Mouse</h2>
<p>Actually I rarely use a mouse since I feel at home in the terminal and use a
few programs with vim-like key bindings, like the Pentadactyl extension for
Firefox. Still I sometimes like to have a mouse and I want it to fulfill two
purposes:</p>
<ol>
<li>Regular use next to keyboard, for example for casual gaming</li>
<li>As a remote control from my couch for watching movies</li>
</ol>
<p>The first is easy to fulfill of course, you could even buy 2 cheap mice from
eBay for 1 € including shipping. The second purpose requires a wireless mouse
that works well on any surface, a perfect application for the <a href="https://secure.logitech.com/en-us/product/mx-anywhere2">Logitech
Anywhere MX2</a>. It even
feels a bit like a small version of my old MX518, quite a nice bonus.</p>
<p><img src="/public/hardware2/logitech-anywhere-mx2.jpg" alt="Logitech Anywhere MX2" /></p>
<p>To actually use the mouse as a remote control for movie watching I had to set a
few mouse button binds in <code class="language-plaintext highlighter-rouge">mpv</code>, my video player:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MOUSE_BTN5 run "mixer" "pcm" "-2"
MOUSE_BTN6 run "mixer" "pcm" "+2"
MOUSE_BTN1 cycle sub-visibility
MOUSE_BTN7 add chapter -1
MOUSE_BTN8 add chapter 1
</code></pre></div></div>
<p>Now I can move through the movie manually or by chapter, switch through audio
tracks, enable subtitles and switch through them, change the global system
volume, toggle fullscreen, and pause and play. That’s quite enough for me.</p>
<p>Since I barely touch the mouse I expect the battery to last forever. And since
my mainboard already supports Bluetooth, I used that instead of the dongle.</p>
<p>Unfortunately you have to <a href="https://github.com/mikolajb/skylake-on-linux#bluetooth">extract the Bluetooth
firmware</a> from the
Windows driver.</p>
<h2 id="power-consumption">Power Consumption</h2>
<p>Initially I was worried about the 150 W PSU of the In Win Chopin. But after
some measurements my concerns turned out to be unnecessary. Since I undervolt
my CPU instead of overclocking it, the maximum I managed to achieve by putting
high loads on CPU and IGP at the same time was 90 W, measured at the wall. At
regular usage the system needs about 16 W, which I consider to be a pretty fine
result for such a powerful machine.</p>
<h2 id="final-result">Final Result</h2>
<p>The final build assembled in its new environment, where I’m sitting right now
on a Sunday morning to write this:</p>
<p><img src="/public/hardware2/finished1.jpg" alt="Finished build #1" /></p>
<p>Closeup of the computer case:</p>
<p><img src="/public/hardware2/finished2.jpg" alt="Finished build #2" /></p>
Broken Hardware, Fixes and Hacks over 8 Years2016-06-21T00:00:00+02:00https://hookrace.net/blog/broken-hardware-fixes-hacks-8-years
<p>After reading the feedback of my <a href="/blog/ddnet-evolution-architecture-technology/">recent article about running
DDNet</a>, I noticed that people
found it interesting how I’m trying to minimize money and resources. I also
noticed that I had been doing something similar with my personal computing
hardware setup for an even longer time.</p>
<p>I’ve mostly been using the same hardware for personal computation purposes over
the last 8 years. In this article I want to talk about some of the hardware
I’ve been using, how it broke and how I fixed the problems or worked around
them. My goal was to be frugal about hardware, to keep using the same hardware
for a long time and repair it when possible instead of simply buying new
hardware.</p>
<!--more-->
<p>The reason for this post is that I’m moving and abandoning my old hardware
setup. There may be some interesting tales in here. Depending on how you value
your time it is probably cheaper to simply buy new stuff instead of repairing
old ones, but I consider it a fun activity and a much more rewarding
experience.</p>
<p>Whether you call it planned obsolescence or just cheap manufacturing, it’s a
fact that electronic hardware tends to break rather quickly. Most reviews don’t
take any note of this and so it is difficult to find out which hardware to buy
if you value durability. The best I can do is tell you what you should and
should not have bought 8 years ago.</p>
<p>For each of these topics there are probably much more detailed reports and
guides online, so I will keep it short and just give an overview. If you’re
interested in fixing your own hardware, a search engine of your choice is your
friend.</p>
<h2 id="tools-used">Tools used</h2>
<p><img src="/public/hardware/weller-wecp-20.jpg" alt="Weller WECP-20" /></p>
<p>The tools I use are absolutely simple: Mostly I just used a Weller WECP-20 for
soldering, regular screwdrivers and some other, older hardware to steal
replacement parts from. Nothing fancy and I don’t claim to do anything special
with it.</p>
<h2 id="liquid-crystal-displays">Liquid-Crystal Displays</h2>
<p>Let’s start with what in my experience was the most common problem in consumer
electronics, resulting in their failure: broken capacitors</p>
<p>Or at least it was a major problem for me, probably because much of my failing
hardware was produced around the time of the <a href="https://en.wikipedia.org/wiki/Capacitor_Plague">capacitor
plague</a>.</p>
<p><img style="max-width: 58%; display: inline; padding: 0;" src="/public/hardware/bad-caps.jpg" alt="Various bad capacitors" /><img style="max-width: 40%; display: inline; padding: 0 0 0 1mm;" src="/public/hardware/bad-caps-2.jpg" alt="Another bulging capacitor" /></p>
<p>On the right picture you can see difference between a good capacitor (left)
and a bulging capacitor (right).</p>
<p>I used a BENQ FP91GP display for the entire time. At some point I also added a
used Samsung SyncMaster P2250.</p>
<p><img class="halfimg" src="/public/hardware/fp91gp.jpg" alt="BENQ FP91GP" /><img class="halfimg" src="/public/hardware/samsung-syncmaster-p2250.jpg" alt="Samsung SyncMaster P2250" /></p>
<p>Both of them failed at some point, stopping to display anything and making
high-pitched sounds instead. So of course the first thing I did was open them
up. In both of them I found the exact same thing: Bulging capacitors</p>
<p>When you open up hardware you have to be careful about electric shocks. I never
received one, but especially power supply units (PSUs), which were the broken
parts inside of my displays, can store a high amount of energy for a long time
after disconnecting them from power.</p>
<p>I didn’t want to buy new capacitors, even though they are available for low
prices and you can even buy a full repair kit online for commonly failing
displays. Instead I went through even older hardware in the house and
scavenged still functioning capacitors. Mostly you have to look out for three
things:</p>
<ol>
<li>New capacitor has <a href="https://web.archive.org/web/20160204013821/https://www.niccomp.com/help/capsubguide.asp">correct characteristics</a></li>
<li>New capacitor fits into space of the old one</li>
<li>Install the new capacitor the right way or more hardware might blow up
(negative stripe on capacitor to white mark on board)</li>
</ol>
<p>Once I found replacement capacitors I desoldered the broken ones and replaced
them. Surprisingly these capacitors from TVs from the 80s still work just fine.
My plan was to buy brand-new capacitors once they fail, but it never happened.</p>
<p>In hindsight it’s certainly curious to repair old hardware with parts from even
older hardware.</p>
<h2 id="power-supply-units">Power Supply Units</h2>
<p>The same story didn’t just happen with the PSUs inside of both of my displays,
but also with the power supply unit of my computer. I fixed it in the same way
and the PSU still works to this day. But since my LC Power LC6420 was never
very efficient I bought a replacement anyway, the more efficient HuntKey Jumper
300G.</p>
<p><img class="halfimg" src="/public/hardware/lc-power-lc-6420.jpg" alt="LC Power LC 6420" /><img class="halfimg" src="/public/hardware/lc-power-lc-6420-2.jpg" alt="Inside of the LC 6420" /></p>
<p>Unfortunately the new PSU turned out to be too loud for my taste, so I opened
it up immediately and noticed that the fan was fixed to 12 V. I reconnected it
to 5 V instead and tested that the PSU does not overheat with my system even at
high load. I wouldn’t recommend doing this since it immediately voids the
warranty and the PSU can overheat easily. Ideally just buy a modern PSU which
controls its fan properly based on the internal temperature.</p>
<h2 id="computer-case">Computer Case</h2>
<p><img class="halfimg" src="/public/hardware/nzxt-zero.jpg" alt="NZXT Zero case" /><img class="halfimg" src="/public/hardware/molex.jpg" alt="Molex extension cable" /></p>
<p>I wasn’t just annoyed with the noise of my PSU though. In the same vein I also
switched all 7 case fans of my NZXT Zero from 12 V to 5 V, thus reducing the
amount of noise significantly. The fix is simple and does not require any
soldering. Instead I used a molex extension cable for the fans and switched its
12 V line to 5 V</p>
<p>The trick here is to remove the pins from the molex cable with a small
screwdriver, and switch them around so that 12 V and 5 V are switched, which is
quite simple. I just had to make sure that my computer stays cool enough, but
with 9 fans in total running inside of it that was no problem, even in the
nearly noiseless state of running at 5 V.</p>
<p>The CPU cooler can be controlled by <code class="language-plaintext highlighter-rouge">fancontrol</code> on Linux and only spins up
when the load is high, otherwise the CPU can be cooled passively, just using
the slight flow of air from the case fans.</p>
<h2 id="random-access-memory">Random Access Memory</h2>
<p><img src="/public/hardware/ram-ddr3.jpg" alt="DDR3 RAM" /></p>
<p>Finally let’s get to an entirely different problem, for which I didn’t even
suspect a hardware fault: I’ve been using Gentoo for nearly the entire time,
which means I ended up compiling my own programs and libraries. Compiling a single
large-scale C++ library failed again and again, at seemingly random positions.
The g++ compiler always complained about internal compiler errors. At first I
assumed this was a bug in the GNU Compiler Collection, so I switched to other
versions, but got the same error in each of them. At some point I noticed that
the memory usage increases to multiple GBs when compiling this library and
that’s when it hit me: My main memory might be defective and with most programs
I just don’t notice.</p>
<p>I booted into Memtest86+ to run the memory benchmark and, lo and behold, a
single byte of my RAM was broken. These were my choices on how to fix this:</p>
<ol>
<li>Remove the RAM module, but then I don’t get the speed benefit of dual
channel and am missing out on 2 GB of RAM</li>
<li>Buy a replacement module, but that costs money</li>
<li>Tell the Linux kernel to ignore the broken part of the memory and use the rest of the module</li>
</ol>
<p>Of course I went with choice 3. A simple Linux kernel parameter
<code class="language-plaintext highlighter-rouge">memmap=1$0x0007cec2d74</code> in GRUB marks 1 byte at the address <code class="language-plaintext highlighter-rouge">0x0007cec2d74</code> as
reserved (<code class="language-plaintext highlighter-rouge">$</code>), preventing the system from using it as regular main memory. A
reboot later everything worked again and I could compile the C++ library. It’s
possible that further parts of this memory module will break in the future and
you have to repeat this process to disable them as well, but so far this has
not happened.</p>
<p>Working around hardware problems with software is cool. And it’s not like I
notice a single byte of memory missing.</p>
<h2 id="graphics-processing-units">Graphics Processing Units</h2>
<p><img class="halfimg" src="/public/hardware/geforce-8500-gt.jpg" alt="GeForce 8500 GT" /><img class="halfimg" src="/public/hardware/radeon-hd4350.jpg" alt="Radeon HD 4350" /></p>
<p>My graphics cards like to break. I don’t really play much except for some 2D
games, so a good GPU never mattered to me. The GeForce 8500 GT started to show
graphics errors after a few years, even while booting, which led me to believe
that the problem was on the hardware side and not software related.</p>
<p>Putting the GPU into an oven and baking it for a while fixed the problem for a
few days by resoldering cracked solder points, but then it returned. This is
probably related to the switch to lead-free solder in 2006, which is more
brittle.</p>
<p>I didn’t want to keep baking my GPU, so I used another GPU I had lying around,
which someone else had thrown away: A Radeon HD 4350</p>
<h2 id="mouse">Mouse</h2>
<p><img src="/public/hardware/logitech-mx-518.jpg" alt="Logitech MX518" /></p>
<p>The Logitech MX518 did its job great for the entire time. About 1-2 years ago
the left mouse button stopped working reliably though. When keeping it pressed
it sometimes released on its own at unpredictable moments. That’s a huge
problem while playing something like <a href="https://ddnet.org/">DDNet</a>.</p>
<p>The first step was of course to open up the mouse. What I saw was that the
plastic of the mouse buttons was worn out to the point that it could barely
press down the actual button inside of the mouse. I reinforced both mouse
buttons with a small piece of thin cardboard. Since then the mouse has not made
any further problems and the buttons work absolutely reliably.</p>
<h2 id="keyboard">Keyboard</h2>
<p><img src="/public/hardware/logitech-ultra-flat-x.jpg" alt="Logitech Ultra Flat-X" /></p>
<p>I’m using a cheap Logitech Ultra Flat-X keyboard, which doesn’t feel terribly
different from a good Thinkpad keyboard. I replaced the green LEDs with blue
ones, but that was just an aesthetic change and was easily done with some more
soldering.</p>
<p>Unfortunately dust and dirt always accumulate under the keys of a keyboard. If
this keeps going for long enough some keys become unusable, so I took out all
the key caps every year or so to wash them and clean the space below them.
Unfortunately the scissor mechanism on cheap keyboards is quite fragile, so you
have to be careful not to break your keys. I don’t need a numpad, so I used the
numpad keys as replacements.</p>
<p><img src="/public/hardware/keycaps.jpg" alt="Keyboard with keys sanded off" /></p>
<p>After a few years of usage the key labels wear off. So instead I sanded the
key labels off entirely, leaving clear key caps which look much better even
after many more years of usage. Unfortunately by now the key caps have become
so thin that they are starting to break apart, so it’s definitely time for a new
keyboard.</p>
<h2 id="printer">Printer</h2>
<p><img style="max-width: 70%; display: inline; padding: 0;" src="/public/hardware/lj4l.jpg" alt="HP LaserJet 4L" /><img style="max-width: 30%; display: inline; padding: 0 0 0 1mm;" src="/public/hardware/pickup-roller.jpg" alt="Pickup Roller" /></p>
<p>This is the oldest piece of hardware in the list so far. This HP LaserJet 4L
has been in continuous use for about 23 years. The only serious thing you need
to worry about with it is to have fresh toner cartridges if you plan to print
extensively. Luckily I got 10 full toner cartridges on eBay for 1 €, taking
care of this problem.</p>
<p>The only part of the printer I had to replace was the pickup roller, which
became unable to pull in any pages. I tried cleaning it, but in the end decided
to just buy a cheap replacement part, which is easy to find in online shops,
even 23 years after the introduction of the printer.</p>
<h2 id="headphones">Headphones</h2>
<p><img style="max-width: 40%; display: inline; padding: 0;" src="/public/hardware/superlux-hd-330.jpg" alt="Superlux HD-330" /><img style="max-width: 60%; display: inline; padding: 0 0 0 1mm;" src="/public/hardware/beyerdynamic-ear-pads.jpg" alt="Beyerdynamic Ear Pads" /></p>
<p>My headphones were not really broken, but rather are a chimera. The base are
cheap Superlux HD-330 headphones (29 €), a not-so-subtle clone of the
Beyerdynamic DT-770 Pro (137 €).</p>
<p>Nevertheless they still sound fine to me, but I don’t claim to be an expert.
Since I didn’t like the cable I used the old nylon cable from my previous
headphones and soldered them in as replacements. A little bit of soldering gets
you quite far.</p>
<p>Instead of using the included fake leather ear pads I bought the original ear
pads from Beyerdynamic (21 €), which cost nearly as much as the rest of the
headphones.</p>
<h2 id="amplifier">Amplifier</h2>
<p><img style="max-width: 70%; display: inline; padding: 0;" src="/public/hardware/marantz-pm710dc.jpg" alt="Marantz PM710DC" /><img style="max-width: 30%; display: inline; padding: 0 0 0 1mm;" src="/public/hardware/heco-superior-700.jpg" alt="Heco Superior 700" /></p>
<p>Finally here’s the oldest part of my setup: A Marantz PM710DC amplifier from
1980 and HECO Superior 700 speakers from around 1985. The speakers still sound
just fine. The amplifier on the other hand is showing its age:</p>
<p>The analog potentiometers are filled with dust and you hear a crackling sound
when adjusting the volume. Blowing on them with some compressed air and moving
the potentiometer a few times entirely removes the crackling for me. In more
serious cases potentiometers can also be cleaned with contact cleaner or simply
be replaced with some soldering.</p>
<h2 id="desk-light">Desk Light</h2>
<p><img src="/public/hardware/ikea-global-work-lamp.jpg" alt="IKEA Global in working state" /></p>
<p>The IKEA Global work lamp is quite useful, as it can be attached to many
surfaces with a clamp and can be moved flexibly. Unfortunately all the flexible
movement kept weakening the plastic connection between lamp and metal arm. In the
end the plastic simply snapped off. My plastic glue didn’t work with this kind
of plastic, so I decided to work around the problem instead.</p>
<p>Using a few pieces of paper as a buffer the lamp can be inserted into the clamp
directly. Now I have the lamp fixed directly above the desk instead of being
able to move it around freely, but that’s not a big problem because the entire
desk is still well-lit.</p>
<h2 id="network-cables">Network Cables</h2>
<p><img style="max-width: 33%; display: inline; padding: 0;" src="/public/hardware/cat5e-roll.jpg" alt="100 m CAT5e roll" /><img style="max-width: 33%; display: inline; padding: 0;" src="/public/hardware/rj45-jack.jpg" alt="RJ45 Jack" /><img style="max-width: 33%; display: inline; padding: 0;" src="/public/hardware/crimping-tool.jpg" alt="Crimping Tool" /></p>
<p>I used to have lots of old Ethernet cables, some working only up to 100 Mbit/s,
others not working at all at times. Instead of buying many new Ethernet cables,
I opted to buy a 100 meter roll of high quality CAT5e Ethernet cable, a
hundred RJ45 jacks and a crimper tool. Finally I can build as many cables as I
need with the exact lengths I prefer.</p>
<p><img src="/public/hardware/cables.jpg" alt="Using newly made network cables" /></p>
<h2 id="laptops">Laptops</h2>
<p><img class="halfimg" src="/public/hardware/t43p.jpg" alt="Thinkpad T43p" /><img class="halfimg" src="/public/hardware/t43p-fan.jpg" alt="Thinkpad T43p Fan" /></p>
<p>Initially I had a used Thinkpad T43p from 2005. After a few years of running
Gentoo on it and compiling too many packages the fan broke. Fortunately
replacement fans are easy to find and opening up and repairing old Thinkpads is
a blessing. Printed on the bottom of the laptop you are instructed how to open
which part of the laptop. There are official guides for replacing any part of
the laptop.</p>
<p><img class="halfimg" src="/public/hardware/x200s.jpg" alt="Thinkpad x200s" /><img class="halfimg" src="/public/hardware/x200s-battery.jpg" alt="Thinkpad x200s Battery" /></p>
<p>Since 2010 I’m running a used Thinkpad x200s. The only problem it ever had was
a broken battery. But since the laptop features replaceable batteries installing
a cheap replacement battery from eBay is a matter of seconds.</p>
<h2 id="home-servers">Home Servers</h2>
<p><img class="halfimg" src="/public/hardware/server.jpg" alt="Thinkpad T42 Server" /><img class="halfimg" src="/public/q1900-itx.jpg" alt="Q1900-ITX Server" /></p>
<p>Broken Thinkpads make good servers. I had an old, very broken Thinkpad T42
lying around. The screen did not work and it could only be powered from the
battery, while it didn’t run on AC power at all and couldn’t charge the battery
either. My solution was to remove the screen entirely and build a small adapter
with some soldering to connect AC power to the battery connection directly,
leaving out the battery.</p>
<p>This worked as a home server for a few years, until the performance of such an
old Pentium M system got too low and I switched to a totally passive Q1900-ITX
system in 2014.</p>
<h2 id="conclusion">Conclusion</h2>
<p>All in all I would say that the hardware held up reasonably well and when it
broke it was usually possible to fix it without replacing the entire device.
More repairable hardware would still be a huge bonus, especially since the
industry has been moving in the opposite direction with ever smaller and more
compacted devices.</p>
<p>Maybe someone who read this post found it interesting and will have fun fixing
their own hardware (or trying to) once it starts malfunctioning.</p>
<p>I wrote this up on the exact hardware described in this post, while waiting for
my new hardware to arrive in the next days. I hope build quality and
repairability of PC hardware haven’t gone down in the last years. But only time
will tell.</p>
<p>Discussion on <a href="https://news.ycombinator.com/item?id=11942618">Hacker News</a> and <a href="https://www.reddit.com/r/hardware/comments/4pap8e/broken_hardware_fixes_and_hacks_over_8_years/">r/Hardware</a>.</p>
Writing a 2D Platform Game in Nim with SDL22016-06-14T00:00:00+02:00https://hookrace.net/blog/writing-a-2d-platform-game-in-nim-with-sdl2
<p><a href="http://postd.cc/writing-a-2d-platform-game-in-nim-with-sdl2/">Japanese Translation</a></p>
<p>In this article we’re going to write a simple 2D <a href="https://en.wikipedia.org/wiki/Platform_game">platform game</a>. You can also consider this as a tutorial for game development with SDL2 in Nim.</p>
<p>We will read in user input, display graphics and a tile map, and simulate
simple 2D physics with collision detection and handling. Afterwards we will
implement simple camera movement and game logic. To display some information we
will render texts and develop a caching mechanism for said text rendering.</p>
<p>The final result will be a binary file that requires only SDL2 and can be
easily distributed, perfect for games. If you’re on Linux we will also present
a simple way to cross-compile Nim programs for Windows.</p>
<!--more-->
<p>For the sake of simplicity we’re going to use the familiar graphics from
<a href="https://ddnet.org/">DDNet</a> and <a href="https://www.teeworlds.com/">Teeworlds</a>, with
the end result of this tutorial looking like this:</p>
<video controls="" muted="" poster="/public/platformer/video-preview3.png">
<source src="/public/platformer/platformer-finished.mp4" type="video/mp4" />
</video>
<p>We’re going to follow along with the development throughout this article with
illustrative images and videos, but the best way to learn is if you follow
along yourself by implementing the steps from this article. The code is
purposefully kept simple and easy to extend so that you can play around with it
and try out all kinds of changes to get an intuitive understanding. At the end
of every section there is a link to its full source code.</p>
<p>The iterations of the code of this article and the final result are available in a <a href="https://github.com/def-/nim-platformer">repository on GitHub</a>. The resulting binaries can be downloaded here: <a href="/public/platformer/platformer_1.0_win64.zip">Win64</a>, <a href="/public/platformer/platformer_1.0_win32.zip">Win32</a>, <a href="/public/platformer/platformer_1.0_linux_x86_64.tar.gz">Linux x86_64</a>, <a href="/public/platformer/platformer_1.0_linux_x86.tar.gz">Linux x86</a></p>
<h2 id="preliminaries">Preliminaries</h2>
<p>For this post we require:</p>
<ul>
<li>The <a href="http://nim-lang.org/">Nim</a> programming language and its package manager <a href="https://github.com/nim-lang/nimble">Nimble</a></li>
<li><a href="https://www.libsdl.org/">SDL 2</a>, <a href="https://www.libsdl.org/projects/SDL_image/">SDL_image 2</a>, <a href="https://www.libsdl.org/projects/SDL_ttf/">SDL_ttf 2</a> (all for developers)</li>
<li><a href="https://github.com/nim-lang/sdl2">Nim SDL2 wrapper</a> and <a href="http://lyro.bitbucket.org/strfmt/">strfmt</a>, which can be installed using Nimble</li>
</ul>
<p>On a unixoid system like Linux or Mac OS X the installation looks something
like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Debian / Ubuntu
$ sudo apt-get install git libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev
# Arch Linux
$ pacman -S git sdl2 sdl2_image sdl2_ttf
# Homebrew on OS X
$ brew install git sdl2 sdl2_image sdl2_ttf
# FreeBSD
$ pkg install git sdl2 sdl2_image sdl2_ttf
$ wget http://nim-lang.org/download/nim-0.14.2.tar.xz
$ tar xvf nim-0.14.2.tar.xz
$ cd nim-0.14.2
$ make -j4
$ echo 'export PATH=$HOME/nim-0.14.2/bin:$PATH' >> ~/.profile
$ source ~/.profile
$ git clone https://github.com/nim-lang/nimble.git
$ cd nimble
$ nim -d:release c -r src/nimble install
$ echo 'export PATH=$HOME/.nimble/bin:$PATH' >> ~/.profile
$ source ~/.profile
$ nimble install sdl2 strfmt
</code></pre></div></div>
<p>Note that you also need a C compiler on your system, preferably GCC or Clang.</p>
<p>Instead of compiling Nim and Nimble from source code you could also use your
package manager to install Nim and Nimble if they’re available in a recent
version.</p>
<p>For setting up SDL2 on other platforms <a href="http://lazyfoo.net/tutorials/SDL/01_hello_SDL/index.php">more extensive guides</a> exist, as do for <a href="http://nim-lang.org/download.html">Nim</a> and <a href="https://github.com/nim-lang/nimble#installation">nimble</a>.</p>
<h2 id="1-first-running-program">1. First Running Program</h2>
<p>If you have seen SDL2 programs written in C or C++ before, you will notice that
what we’re doing in Nim is very similar. Actually Nim’s SDL2 wrapper is just a
thin layer wrapping the original SDL2 interface from C. This has the advantage
that what you learn from any SDL2 tutorial is applicable, but the disadvantage
is that you end up with a bit more boilerplate than with a more high-level Nim
library.</p>
<p>We start with exactly this boilerplate to initialize our window:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">sdl2</span>
<span class="k">type</span> <span class="n">SDLException</span> <span class="o">=</span> <span class="k">object</span> <span class="k">of</span> <span class="n">Exception</span>
<span class="k">template</span> <span class="n">sdlFailIf</span><span class="p">(</span><span class="n">cond</span><span class="p">:</span> <span class="n">typed</span><span class="p">,</span> <span class="n">reason</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">cond</span><span class="p">:</span> <span class="k">raise</span> <span class="n">SDLException</span><span class="p">.</span><span class="n">newException</span><span class="p">(</span>
<span class="n">reason</span> <span class="o">&</span> <span class="s">", SDL error: "</span> <span class="o">&</span> <span class="o">$</span><span class="n">getError</span><span class="p">())</span>
<span class="k">proc </span><span class="nf">main</span> <span class="o">=</span>
<span class="n">sdlFailIf</span><span class="p">(</span><span class="ow">not</span> <span class="n">sdl2</span><span class="p">.</span><span class="n">init</span><span class="p">(</span><span class="n">INIT_VIDEO</span> <span class="ow">or</span> <span class="n">INIT_TIMER</span> <span class="ow">or</span> <span class="n">INIT_EVENTS</span><span class="p">)):</span>
<span class="s">"SDL2 initialization failed"</span>
<span class="c"># defer blocks get called at the end of the procedure, even if an</span>
<span class="c"># exception has been thrown</span>
<span class="k">defer</span><span class="p">:</span> <span class="n">sdl2</span><span class="p">.</span><span class="n">quit</span><span class="p">()</span>
<span class="n">sdlFailIf</span><span class="p">(</span><span class="ow">not</span> <span class="n">setHint</span><span class="p">(</span><span class="s">"SDL_RENDER_SCALE_QUALITY"</span><span class="p">,</span> <span class="s">"2"</span><span class="p">)):</span>
<span class="s">"Linear texture filtering could not be enabled"</span>
<span class="k">let</span> <span class="n">window</span> <span class="o">=</span> <span class="n">createWindow</span><span class="p">(</span><span class="n">title</span> <span class="o">=</span> <span class="s">"Our own 2D platformer"</span><span class="p">,</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">SDL_WINDOWPOS_CENTERED</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">SDL_WINDOWPOS_CENTERED</span><span class="p">,</span>
<span class="n">w</span> <span class="o">=</span> <span class="mi">1280</span><span class="p">,</span> <span class="n">h</span> <span class="o">=</span> <span class="mi">720</span><span class="p">,</span> <span class="n">flags</span> <span class="o">=</span> <span class="n">SDL_WINDOW_SHOWN</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">window</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Window could not be created"</span>
<span class="k">defer</span><span class="p">:</span> <span class="n">window</span><span class="p">.</span><span class="n">destroy</span><span class="p">()</span>
<span class="k">let</span> <span class="n">renderer</span> <span class="o">=</span> <span class="n">window</span><span class="p">.</span><span class="n">createRenderer</span><span class="p">(</span><span class="n">index</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span>
<span class="n">flags</span> <span class="o">=</span> <span class="n">Renderer_Accelerated</span> <span class="ow">or</span> <span class="n">Renderer_PresentVsync</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">renderer</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Renderer could not be created"</span>
<span class="k">defer</span><span class="p">:</span> <span class="n">renderer</span><span class="p">.</span><span class="n">destroy</span><span class="p">()</span>
<span class="c"># Set the default color to use for drawing</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">setDrawColor</span><span class="p">(</span><span class="n">r</span> <span class="o">=</span> <span class="mi">110</span><span class="p">,</span> <span class="n">g</span> <span class="o">=</span> <span class="mi">132</span><span class="p">,</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">174</span><span class="p">)</span>
<span class="c"># Game loop, draws each frame</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="c"># Draw over all drawings of the last frame with the default</span>
<span class="c"># color</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="c"># Show the result on screen</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">present</span><span class="p">()</span>
<span class="n">main</span><span class="p">()</span></code></pre></figure>
<p>We introduced an <code class="language-plaintext highlighter-rouge">sdlFailIf</code> template that checks a condition and if the
condition is true, raises an <code class="language-plaintext highlighter-rouge">SDLException</code> with additional error information
from SDL. In the <code class="language-plaintext highlighter-rouge">main</code> proc we initialize SDL2, and create a regular window
and an accelerated 2D renderer. Error handling is done with the <code class="language-plaintext highlighter-rouge">sdlFailIf</code>
proc that we introduced.</p>
<p>For now the game loop just clears the window and draws it every frame. If you
have VSync enabled and your screen is set to 60 Hz the loop will be executed 60
times per second.</p>
<p>We can compile and run in the same step by executing <code class="language-plaintext highlighter-rouge">nim -r c platformer</code>,
assuming you called the file <code class="language-plaintext highlighter-rouge">platformer.nim</code>. To compile with optimizations
use <code class="language-plaintext highlighter-rouge">nim -d:release -r c platformer</code>. The result is a simple one-colored
window:</p>
<p><img src="/public/platformer/platformer-empty.png" alt="Just a window with blue content" /></p>
<p>We can exit our small program by pressing Ctrl-C in the terminal window.
Unfortunately we can’t exit it in the game window itself yet, so let’s fix
that.</p>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part1.nim">Full code for section 1</a></p>
<h2 id="2-user-input">2. User Input</h2>
<p>First let’s add an <code class="language-plaintext highlighter-rouge">Input</code> type to store all the inputs we want to support, and
store an array of Inputs in our game state object:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">type</span>
<span class="n">Input</span> <span class="p">{.</span><span class="n">pure</span><span class="p">.}</span> <span class="o">=</span> <span class="k">enum</span> <span class="n">none</span><span class="p">,</span> <span class="n">left</span><span class="p">,</span> <span class="n">right</span><span class="p">,</span> <span class="n">jump</span><span class="p">,</span> <span class="n">restart</span><span class="p">,</span> <span class="n">quit</span>
<span class="n">Game</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">inputs</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="n">Input</span><span class="p">,</span> <span class="kt">bool</span><span class="o">]</span>
<span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span></code></pre></figure>
<p>By choosing a <code class="language-plaintext highlighter-rouge">ref</code> type for the <code class="language-plaintext highlighter-rouge">Game</code> state type we have an easy way to
prevent accidentally creating copies of it. By default only the garbage
collected pointer to our Game object is passed around. The <code class="language-plaintext highlighter-rouge">inputs</code> field is an
array mapping from <code class="language-plaintext highlighter-rouge">Input</code> to <code class="language-plaintext highlighter-rouge">bool</code>, signifying which input is currently
pressed (<code class="language-plaintext highlighter-rouge">true</code>) or not (<code class="language-plaintext highlighter-rouge">false</code>). Creating a new game state object is trivial,
we just create a new heap object for now and assign the SDL2 <code class="language-plaintext highlighter-rouge">renderer</code> that we
will need later:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">newGame</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">):</span> <span class="n">Game</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">renderer</span> <span class="o">=</span> <span class="n">renderer</span></code></pre></figure>
<p>We don’t need to initialize the <code class="language-plaintext highlighter-rouge">inputs</code> field in any way as everything is
initialized to binary null by default, which is exactly what we want: Every
input is set to off in the start. If we didn’t initialize the <code class="language-plaintext highlighter-rouge">renderer</code> field
it would be a null pointer and we would get into trouble if we accidentally
dereference it.</p>
<p>The next thing we need is a procedure that maps keyboard scan codes to our
recognized inputs:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">toInput</span><span class="p">(</span><span class="n">key</span><span class="p">:</span> <span class="n">Scancode</span><span class="p">):</span> <span class="n">Input</span> <span class="o">=</span>
<span class="k">case</span> <span class="n">key</span>
<span class="k">of</span> <span class="n">SDL_SCANCODE_A</span><span class="p">:</span> <span class="n">Input</span><span class="p">.</span><span class="n">left</span>
<span class="k">of</span> <span class="n">SDL_SCANCODE_D</span><span class="p">:</span> <span class="n">Input</span><span class="p">.</span><span class="n">right</span>
<span class="k">of</span> <span class="n">SDL_SCANCODE_SPACE</span><span class="p">:</span> <span class="n">Input</span><span class="p">.</span><span class="n">jump</span>
<span class="k">of</span> <span class="n">SDL_SCANCODE_R</span><span class="p">:</span> <span class="n">Input</span><span class="p">.</span><span class="n">restart</span>
<span class="k">of</span> <span class="n">SDL_SCANCODE_Q</span><span class="p">:</span> <span class="n">Input</span><span class="p">.</span><span class="n">quit</span>
<span class="k">else</span><span class="p">:</span> <span class="n">Input</span><span class="p">.</span><span class="n">none</span></code></pre></figure>
<p>Note that <code class="language-plaintext highlighter-rouge">toInput</code> returns <code class="language-plaintext highlighter-rouge">Input.none</code> for all undefined cases. We will use
this behaviour to ignore unused keyboard inputs without the need for a branch
in our code. You could easily recognize multiple scan codes to map to a single
input.</p>
<p>We modify the game loop to react to keyboard inputs by calling our new
<code class="language-plaintext highlighter-rouge">handleInput</code> proc. We also split out the rendering itself to keep the
separation of concerns clear in the <code class="language-plaintext highlighter-rouge">main</code> proc:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">handleInput</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">event</span> <span class="o">=</span> <span class="n">defaultEvent</span>
<span class="k">while</span> <span class="n">pollEvent</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
<span class="k">case</span> <span class="n">event</span><span class="p">.</span><span class="n">kind</span>
<span class="k">of</span> <span class="n">QuitEvent</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">quit</span><span class="o">]</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">of</span> <span class="n">KeyDown</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">event</span><span class="p">.</span><span class="n">key</span><span class="p">.</span><span class="n">keysym</span><span class="p">.</span><span class="n">scancode</span><span class="p">.</span><span class="n">toInput</span><span class="o">]</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">of</span> <span class="n">KeyUp</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">event</span><span class="p">.</span><span class="n">key</span><span class="p">.</span><span class="n">keysym</span><span class="p">.</span><span class="n">scancode</span><span class="p">.</span><span class="n">toInput</span><span class="o">]</span> <span class="o">=</span> <span class="kp">false</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">discard</span>
<span class="k">proc </span><span class="nf">render</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="c"># Draw over all drawings of the last frame with the default color</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="c"># Show the result on screen</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">present</span><span class="p">()</span>
<span class="k">var</span> <span class="n">game</span> <span class="o">=</span> <span class="n">newGame</span><span class="p">(</span><span class="n">renderer</span><span class="p">)</span>
<span class="c"># Game loop, draws each frame</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">quit</span><span class="o">]</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">handleInput</span><span class="p">()</span>
<span class="n">game</span><span class="p">.</span><span class="n">render</span><span class="p">()</span></code></pre></figure>
<p>Now we can either press <em>q</em> or use the <em>close</em> button on the window to close
the game. Other kinds of user input are just stored in the <code class="language-plaintext highlighter-rouge">inputs</code> array for
now and we will be able to use it later.</p>
<p>Note that the inputs array is intentionally a simple array, so that access to
it is as performant as can be. If we used a hash table or some other data
structure such a guarantee would not be that easy to make. Simplicity can often
be beneficial and make it easier to understand what is going on in the system
you’re developing.</p>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part2.nim">Full code for section 2</a></p>
<h2 id="3-displaying-graphics">3. Displaying Graphics</h2>
<p>If we want to do something of interest with those user inputs we need to start
displaying something other than a blue sky.</p>
<p>We extend our game state object to also store the player texture as well as the
current position and velocity, for which we use the <a href="http://nim-lang.org/docs/basic2d.html">basic2d module</a>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">basic2d</span>
<span class="k">type</span>
<span class="n">Player</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span>
<span class="n">pos</span><span class="p">:</span> <span class="n">Point2d</span>
<span class="n">vel</span><span class="p">:</span> <span class="n">Vector2d</span>
<span class="n">Game</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">inputs</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="n">Input</span><span class="p">,</span> <span class="kt">bool</span><span class="o">]</span>
<span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span>
<span class="n">player</span><span class="p">:</span> <span class="n">Player</span>
<span class="n">camera</span><span class="p">:</span> <span class="n">Vector2d</span></code></pre></figure>
<p>We’re going to use Teeworlds’ default tee graphic for our player:</p>
<p><a href="/public/platformer/player.png"><img style="display: inline;" alt="Player" src="/public/platformer/player.png" /><img style="display: inline;" alt="Player" src="/public/platformer/preview-player.png" /></a></p>
<p>You can save this file as <a href="/public/platformer/player.png">player.png</a> to follow along. If
you feel funny you can also select one of hundreds of skins from the <a href="https://ddnet.org/skins/">DDNet
Skin Database</a>, for example:</p>
<p><a href="https://ddnet.org/skins/skin/Apish%20Coke.png"><img style="display: inline;" alt="Apish Coke" src="/public/platformer/preview-Apish%20Coke.png" /></a><a href="https://ddnet.org/skins/skin/aqua.png"><img style="display: inline;" alt="aqua" src="/public/platformer/preview-aqua.png" /></a><a href="https://ddnet.org/skins/skin/cutie.png"><img style="display: inline;" alt="cutie" src="/public/platformer/preview-cutie.png" /></a><a href="https://ddnet.org/skins/skin/dragon.png"><img style="display: inline;" alt="dragon" src="/public/platformer/preview-dragon.png" /></a><a href="https://ddnet.org/skins/skin/hammie-chew.png"><img style="display: inline;" alt="hammie-chew" src="/public/platformer/preview-hammie-chew.png" /></a><a href="https://ddnet.org/skins/skin/penguin.png"><img style="display: inline;" alt="penguin" src="/public/platformer/preview-penguin.png" /></a><a href="https://ddnet.org/skins/skin/tank.png"><img style="display: inline;" alt="tank" src="/public/platformer/preview-tank.png" /></a><a href="https://ddnet.org/skins/skin/turtle_r.png"><img style="display: inline;" alt="turtle_r" src="/public/platformer/preview-turtle_r.png" /></a><a href="https://ddnet.org/skins/skin/bomb.png"><img style="display: inline;" alt="bomb" src="/public/platformer/preview-bomb.png" /></a><a href="https://ddnet.org/skins/skin/coala.png"><img style="display: inline;" alt="coala" src="/public/platformer/preview-coala.png" /></a><a href="https://ddnet.org/skins/skin/Tutawek1.png"><img style="display: inline;" alt="Tutawek1" src="/public/platformer/preview-Tutawek1.png" /></a><a href="https://ddnet.org/skins/skin/zzz.png"><img style="display: inline;" alt="zzz" src="/public/platformer/preview-zzz.png" /></a><a href="https://ddnet.org/skins/skin/robin_hood.png"><img style="display: inline;" alt="robin_hood" src="/public/platformer/preview-robin_hood.png" /></a><a href="https://ddnet.org/skins/skin/red_bird.png"><img style="display: inline;" alt="red_bird" src="/public/platformer/preview-red_bird.png" /></a><a href="https://ddnet.org/skins/skin/dino.png"><img style="display: inline;" alt="dino" src="/public/platformer/preview-dino.png" /></a><a href="https://ddnet.org/skins/skin/godlike.png"><img style="display: inline;" alt="godlike" src="/public/platformer/preview-godlike.png" /></a><a href="https://ddnet.org/skins/skin/lightbulb.png"><img style="display: inline;" alt="lightbulb" src="/public/platformer/preview-lightbulb.png" /></a><a href="https://ddnet.org/skins/skin/mario.png"><img style="display: inline;" alt="mario" src="/public/platformer/preview-mario.png" /></a><a href="https://ddnet.org/skins/skin/mike.png"><img style="display: inline;" alt="mike" src="/public/platformer/preview-mike.png" /></a><a href="https://ddnet.org/skins/skin/mouse.png"><img style="display: inline;" alt="mouse" src="/public/platformer/preview-mouse.png" /></a><a href="https://ddnet.org/skins/skin/Robot.png"><img style="display: inline;" alt="Robot" src="/public/platformer/preview-Robot.png" /></a></p>
<p>Don’t forget to call your player graphic <code class="language-plaintext highlighter-rouge">player.png</code> if you want to use an
alternative one.</p>
<p>First we have to load the player graphic, whichever one you decided to use, in
our definition of <code class="language-plaintext highlighter-rouge">newGame</code> and initialize the <code class="language-plaintext highlighter-rouge">Game</code> and <code class="language-plaintext highlighter-rouge">Player</code> data
structures:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">sdl2</span><span class="o">/</span><span class="n">image</span>
<span class="k">proc </span><span class="nf">restartPlayer</span><span class="p">(</span><span class="n">player</span><span class="p">:</span> <span class="n">Player</span><span class="p">)</span> <span class="o">=</span>
<span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">=</span> <span class="n">point2d</span><span class="p">(</span><span class="mi">170</span><span class="p">,</span> <span class="mi">500</span><span class="p">)</span>
<span class="n">player</span><span class="p">.</span><span class="n">vel</span> <span class="o">=</span> <span class="n">vector2d</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">newPlayer</span><span class="p">(</span><span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span><span class="p">):</span> <span class="n">Player</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">texture</span> <span class="o">=</span> <span class="n">texture</span>
<span class="n">result</span><span class="p">.</span><span class="n">restartPlayer</span><span class="p">()</span>
<span class="k">proc </span><span class="nf">newGame</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">):</span> <span class="n">Game</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">renderer</span> <span class="o">=</span> <span class="n">renderer</span>
<span class="n">result</span><span class="p">.</span><span class="n">player</span> <span class="o">=</span> <span class="n">newPlayer</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture</span><span class="p">(</span><span class="s">"player.png"</span><span class="p">))</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">restartPlayer</code> is used to reset the player to its start position. The
<code class="language-plaintext highlighter-rouge">loadTexture</code> procedure loads the PNG image into memory as an SDL2 texture that
we can store in the <code class="language-plaintext highlighter-rouge">Game</code> object.</p>
<p>We also shouldn’t forget to initialize the SDL2 image module in our <code class="language-plaintext highlighter-rouge">main</code>
proc, similarly to the SDL2 initialization:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">const</span> <span class="n">imgFlags</span><span class="p">:</span> <span class="n">cint</span> <span class="o">=</span> <span class="n">IMG_INIT_PNG</span>
<span class="n">sdlFailIf</span><span class="p">(</span><span class="n">image</span><span class="p">.</span><span class="n">init</span><span class="p">(</span><span class="n">imgFlags</span><span class="p">)</span> <span class="o">!=</span> <span class="n">imgFlags</span><span class="p">):</span>
<span class="s">"SDL2 Image initialization failed"</span>
<span class="k">defer</span><span class="p">:</span> <span class="n">image</span><span class="p">.</span><span class="n">quit</span><span class="p">()</span></code></pre></figure>
<p>We only need support for PNG files, otherwise we could also add JPEG files with
<code class="language-plaintext highlighter-rouge">const imgFlags: cint = IMG_INIT_PNG or IMG_INIT_JPG</code>.</p>
<p>Next our task is to put this together nicely. Of course the intention of the
flexible player images is that parts of the body can move independently, but
for the sake of simplicity we will put them into fixed positions. A simple
addition would be to make the feet move in a rotating motion depending on the
horizontal position of the player. Another addition would be to make the eyes
follow the mouse cursor.</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">renderTee</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">,</span> <span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span><span class="p">,</span>
<span class="n">pos</span><span class="p">:</span> <span class="n">Point2d</span><span class="p">)</span> <span class="o">=</span>
<span class="k">let</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">pos</span><span class="p">.</span><span class="n">x</span><span class="p">.</span><span class="n">cint</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span><span class="p">.</span><span class="n">cint</span>
<span class="k">var</span> <span class="n">bodyParts</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="mi">8</span><span class="p">,</span> <span class="k">tuple</span><span class="o">[</span><span class="n">source</span><span class="p">,</span> <span class="n">dest</span><span class="p">:</span> <span class="n">Rect</span><span class="p">,</span> <span class="n">flip</span><span class="p">:</span> <span class="n">cint</span><span class="o">]]</span> <span class="o">=</span> <span class="o">[</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span><span class="mi">192</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">60</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">48</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># back feet shadow</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">96</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">48</span><span class="p">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">48</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">96</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># body shadow</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span><span class="mi">192</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">36</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">48</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># front feet shadow</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span><span class="mi">192</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">60</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">48</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># back feet</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">96</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">48</span><span class="p">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">48</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">96</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># body</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span><span class="mi">192</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">36</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">48</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># front feet</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">18</span><span class="p">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">21</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">36</span><span class="p">),</span>
<span class="n">SDL_FLIP_NONE</span><span class="p">),</span> <span class="c"># left eye</span>
<span class="p">(</span><span class="n">rect</span><span class="p">(</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">96</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">rect</span><span class="p">(</span> <span class="n">x</span><span class="o">-</span><span class="mi">6</span><span class="p">,</span> <span class="n">y</span><span class="o">-</span><span class="mi">21</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">36</span><span class="p">),</span>
<span class="n">SDL_FLIP_HORIZONTAL</span><span class="p">)</span> <span class="c"># right eye</span>
<span class="o">]</span>
<span class="k">for</span> <span class="n">part</span> <span class="ow">in</span> <span class="n">bodyParts</span><span class="p">.</span><span class="n">mitems</span><span class="p">:</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">copyEx</span><span class="p">(</span><span class="n">texture</span><span class="p">,</span> <span class="n">part</span><span class="p">.</span><span class="n">source</span><span class="p">,</span> <span class="n">part</span><span class="p">.</span><span class="n">dest</span><span class="p">,</span> <span class="n">angle</span> <span class="o">=</span> <span class="mf">0.0</span><span class="p">,</span>
<span class="n">center</span> <span class="o">=</span> <span class="k">nil</span><span class="p">,</span> <span class="n">flip</span> <span class="o">=</span> <span class="n">part</span><span class="p">.</span><span class="n">flip</span><span class="p">)</span></code></pre></figure>
<p>The exact numbers are not so important, they are just how the player is meant
to be put together. With <code class="language-plaintext highlighter-rouge">renderTee</code> we define which body part is drawn at
which position and in which order. Finally each of these body parts is drawn
with the SDL2 renderer using <code class="language-plaintext highlighter-rouge">copyEx</code>.</p>
<p>Now drawing the tee in our game loop is simply a call to <code class="language-plaintext highlighter-rouge">renderTee</code> away:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">render</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="c"># Draw over all drawings of the last frame with the default color</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="c"># Actual drawing here</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderTee</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">texture</span><span class="p">,</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">-</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="c"># Show the result on screen</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">present</span><span class="p">()</span></code></pre></figure>
<p>Finally we have some visual progress again, look at the player floating in the
sky:</p>
<p><img src="/public/platformer/platformer-player.png" alt="Tee on blue background" /></p>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part3.nim">Full code for section 3</a></p>
<h2 id="4-tile-map">4. Tile Map</h2>
<p>Now that we have a working rendering system for the player, we need a map to
play in. This requires us to store a texture as well as a list of tiles:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">type</span>
<span class="n">Map</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span>
<span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">:</span> <span class="kt">int</span>
<span class="n">tiles</span><span class="p">:</span> <span class="kt">seq</span><span class="o">[</span><span class="n">uint8</span><span class="o">]</span>
<span class="n">Game</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">inputs</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="n">Input</span><span class="p">,</span> <span class="kt">bool</span><span class="o">]</span>
<span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span>
<span class="n">player</span><span class="p">:</span> <span class="n">Player</span>
<span class="n">map</span><span class="p">:</span> <span class="n">Map</span>
<span class="n">camera</span><span class="p">:</span> <span class="n">Vector2d</span></code></pre></figure>
<p>Each tile is defined to be a <code class="language-plaintext highlighter-rouge">uint8</code>, which means a value between 0 and 255
inclusively. Conveniently the tileset graphic for from Teeworlds have 16 × 16 =
256 tiles. We will use the grass tileset:</p>
<p><a href="/public/platformer/grass.png"><img src="/public/platformer/grass.png" alt="Grass Tileset" /></a></p>
<p>Download and save this image as <a href="/public/platformer/grass.png">grass.png</a>.</p>
<p>To initialize the <code class="language-plaintext highlighter-rouge">Map</code> data structure we shall write a little parser that
parses maps of this format:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 0 0 0 0 78 0 0 0 0 0 0 0 0 0 0
4 5 0 0 78 0 0 0 0 0 0 0 0 0 0
20 21 0 0 78 0 0 0 0 0 0 0 0 0 0
20 21 0 0 78 0 0 0 0 0 0 0 0 4 5
20 21 0 0 78 0 0 0 0 0 0 0 0 20 21
20 21 0 0 78 0 0 0 0 0 0 0 0 20 21
20 21 0 0 78 0 0 4 5 0 0 0 0 20 21
20 21 0 0 78 0 0 20 21 0 0 0 0 20 21
20 38 0 0 78 0 0 22 38 0 0 0 0 22 38
20 49 16 16 16 16 16 48 49 16 16 16 16 48 49
36 52 52 52 52 52 52 52 52 52 52 52 52 52 52
</code></pre></div></div>
<p>Our goal in this section is for this map with the grass tileset to result in
this rendering:</p>
<p><img src="/public/platformer/platformer-partmap.png" alt="Map rendered with grass tileset" /></p>
<p>Each number denotes the tile that is chosen from the grass tileset. We will use
<a href="/public/platformer/default.map">this default.map</a> for the rest of this article. Our
parser is implemented in <code class="language-plaintext highlighter-rouge">newMap</code> and looks like this:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">strutils</span>
<span class="k">proc </span><span class="nf">newMap</span><span class="p">(</span><span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span><span class="p">,</span> <span class="n">file</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="n">Map</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">texture</span> <span class="o">=</span> <span class="n">texture</span>
<span class="n">result</span><span class="p">.</span><span class="n">tiles</span> <span class="o">=</span> <span class="o">@[]</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">file</span><span class="p">.</span><span class="n">lines</span><span class="p">:</span>
<span class="k">var</span> <span class="n">width</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">line</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="sc">' '</span><span class="p">):</span>
<span class="k">if</span> <span class="n">word</span> <span class="o">==</span> <span class="s">""</span><span class="p">:</span> <span class="k">continue</span>
<span class="k">let</span> <span class="n">value</span> <span class="o">=</span> <span class="n">parseUInt</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">if</span> <span class="n">value</span> <span class="o">></span> <span class="n">uint</span><span class="p">(</span><span class="n">uint8</span><span class="p">.</span><span class="n">high</span><span class="p">):</span>
<span class="k">raise</span> <span class="n">ValueError</span><span class="p">.</span><span class="n">newException</span><span class="p">(</span>
<span class="s">"Invalid value "</span> <span class="o">&</span> <span class="n">word</span> <span class="o">&</span> <span class="s">" in map "</span> <span class="o">&</span> <span class="n">file</span><span class="p">)</span>
<span class="n">result</span><span class="p">.</span><span class="n">tiles</span><span class="p">.</span><span class="n">add</span> <span class="n">value</span><span class="p">.</span><span class="n">uint8</span>
<span class="n">inc</span> <span class="n">width</span>
<span class="k">if</span> <span class="n">result</span><span class="p">.</span><span class="n">width</span> <span class="o">></span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">result</span><span class="p">.</span><span class="n">width</span> <span class="o">!=</span> <span class="n">width</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ValueError</span><span class="p">.</span><span class="n">newException</span><span class="p">(</span>
<span class="s">"Incompatible line length in map "</span> <span class="o">&</span> <span class="n">file</span><span class="p">)</span>
<span class="n">result</span><span class="p">.</span><span class="n">width</span> <span class="o">=</span> <span class="n">width</span>
<span class="n">inc</span> <span class="n">result</span><span class="p">.</span><span class="n">height</span></code></pre></figure>
<p>We split the file by line as well as by word. Each number is parsed to an
unsigned integer and checked whether it is in the valid range of <code class="language-plaintext highlighter-rouge">0..255</code>. The
final width and height are calculated from the line length and number of lines.
Errors in the map data cause an exception to be thrown, quitting our game.</p>
<p>We have to extend <code class="language-plaintext highlighter-rouge">newGame</code> to initialize the map now:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">newGame</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">):</span> <span class="n">Game</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">renderer</span> <span class="o">=</span> <span class="n">renderer</span>
<span class="n">result</span><span class="p">.</span><span class="n">player</span> <span class="o">=</span> <span class="n">newPlayer</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture</span><span class="p">(</span><span class="s">"player.png"</span><span class="p">))</span>
<span class="n">result</span><span class="p">.</span><span class="n">map</span> <span class="o">=</span> <span class="n">newMap</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture</span><span class="p">(</span><span class="s">"grass.png"</span><span class="p">),</span>
<span class="s">"default.map"</span><span class="p">)</span></code></pre></figure>
<p>At this point we have the texture and the tiles of the map. What’s missing is
actually rendering all of this:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">const</span>
<span class="n">tilesPerRow</span> <span class="o">=</span> <span class="mi">16</span>
<span class="n">tileSize</span><span class="p">:</span> <span class="n">Point</span> <span class="o">=</span> <span class="p">(</span><span class="mf">64.</span><span class="n">cint</span><span class="p">,</span> <span class="mf">64.</span><span class="n">cint</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">renderMap</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">,</span> <span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">camera</span><span class="p">:</span> <span class="n">Vector2d</span><span class="p">)</span> <span class="o">=</span>
<span class="k">var</span>
<span class="n">clip</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">y</span><span class="p">)</span>
<span class="n">dest</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">y</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">tileNr</span> <span class="ow">in</span> <span class="n">map</span><span class="p">.</span><span class="n">tiles</span><span class="p">:</span>
<span class="k">if</span> <span class="n">tileNr</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">continue</span>
<span class="n">clip</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">cint</span><span class="p">(</span><span class="n">tileNr</span> <span class="ow">mod</span> <span class="n">tilesPerRow</span><span class="p">)</span> <span class="o">*</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">x</span>
<span class="n">clip</span><span class="p">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">cint</span><span class="p">(</span><span class="n">tileNr</span> <span class="ow">div</span> <span class="n">tilesPerRow</span><span class="p">)</span> <span class="o">*</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">y</span>
<span class="n">dest</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">cint</span><span class="p">(</span><span class="n">i</span> <span class="ow">mod</span> <span class="n">map</span><span class="p">.</span><span class="n">width</span><span class="p">)</span> <span class="o">*</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">camera</span><span class="p">.</span><span class="n">x</span><span class="p">.</span><span class="n">cint</span>
<span class="n">dest</span><span class="p">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">cint</span><span class="p">(</span><span class="n">i</span> <span class="ow">div</span> <span class="n">map</span><span class="p">.</span><span class="n">width</span><span class="p">)</span> <span class="o">*</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">camera</span><span class="p">.</span><span class="n">y</span><span class="p">.</span><span class="n">cint</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">copy</span><span class="p">(</span><span class="n">map</span><span class="p">.</span><span class="n">texture</span><span class="p">,</span> <span class="n">unsafeAddr</span> <span class="n">clip</span><span class="p">,</span> <span class="n">unsafeAddr</span> <span class="n">dest</span><span class="p">)</span></code></pre></figure>
<p>This is similar to <code class="language-plaintext highlighter-rouge">renderTee</code> but uses fixed size parts of the texture. The
texture is cut into tiles of size 64 × 64 pixels with 16 tiles per line. We
iterate over each tile in the map and render the tile <code class="language-plaintext highlighter-rouge">tileNr</code> from our map
texture with. Tile 0 is the air tile and is always empty so we don’t need to
render it, which improves performance as typical maps are in large parts empty.</p>
<p>Finally we have to render our map in the <code class="language-plaintext highlighter-rouge">main</code> proc after rendering the
player, so that it is put on top of the player:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">render</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="c"># Draw over all drawings of the last frame with the default color</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="c"># Actual drawing here</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderTee</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">texture</span><span class="p">,</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">-</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderMap</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">map</span><span class="p">,</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="c"># Show the result on screen</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">present</span><span class="p">()</span></code></pre></figure>
<p>For now we don’t have a moving camera, so we use the static <code class="language-plaintext highlighter-rouge">game.camera</code> to
get a fixed rendering position. But for now we can’t even move, so we should
finally do something with the user input and implement a simple physics model.</p>
<p>The end result of this section features our beautiful map rendering:</p>
<p><img src="/public/platformer/platformer-map.png" alt="Renderer Map" /></p>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part4.nim">Full code for section 4</a></p>
<h2 id="5-physics-and-collisions">5. Physics and Collisions</h2>
<p>For our game physics we decide to have 50 ticks per second. We will only
calculate the next iteration of the game when a new tick has arrived,
independent of whether our game runs at 60 fps or even 240 fps. Let’s add the
ticks to our <code class="language-plaintext highlighter-rouge">main</code> proc:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">times</span>
<span class="k">proc </span><span class="nf">physics</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">discard</span>
<span class="k">var</span>
<span class="n">startTime</span> <span class="o">=</span> <span class="n">epochTime</span><span class="p">()</span>
<span class="n">lastTick</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">quit</span><span class="o">]</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">handleInput</span><span class="p">()</span>
<span class="k">let</span> <span class="n">newTick</span> <span class="o">=</span> <span class="kt">int</span><span class="p">((</span><span class="n">epochTime</span><span class="p">()</span> <span class="o">-</span> <span class="n">startTime</span><span class="p">)</span> <span class="o">*</span> <span class="mi">50</span><span class="p">)</span>
<span class="k">for</span> <span class="n">tick</span> <span class="ow">in</span> <span class="n">lastTick</span><span class="o">+</span><span class="mi">1</span> <span class="p">..</span> <span class="n">newTick</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">physics</span><span class="p">()</span>
<span class="n">lastTick</span> <span class="o">=</span> <span class="n">newTick</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">render</span><span class="p">(</span><span class="n">game</span><span class="p">)</span></code></pre></figure>
<p>That’s our physics framework, but so far the physics does nothing. We can start
by adding gravity:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">physics</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">+=</span> <span class="mf">0.75</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">+=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span></code></pre></figure>
<video controls="" muted="" poster="/public/platformer/video-preview1.png">
<source src="/public/platformer/platformer-gravity.mp4" type="video/mp4" />
</video>
<p>Well, there goes our player. I guess we also need to be able to restart the
player now, so that we can see this amazing animation again and again just by
pressing <em>r</em>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">physics</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">restart</span><span class="o">]</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">restartPlayer</span><span class="p">()</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">+=</span> <span class="mf">0.75</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">+=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span></code></pre></figure>
<p>This looks exactly like the previous GIF on repeat, so you can imagine it or
click on <em>play</em> a few times.</p>
<p>Moving left and right with <em>a</em> and <em>d</em> as well as jumping with <em>space</em> is now
easy to implement:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">physics</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">restart</span><span class="o">]</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">restartPlayer</span><span class="p">()</span>
<span class="k">if</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">jump</span><span class="o">]</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">=</span> <span class="o">-</span><span class="mi">21</span>
<span class="k">let</span> <span class="n">direction</span> <span class="o">=</span> <span class="kt">float</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">right</span><span class="o">]</span><span class="p">.</span><span class="kt">int</span> <span class="o">-</span>
<span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">left</span><span class="o">]</span><span class="p">.</span><span class="kt">int</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">+=</span> <span class="mf">0.75</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span>
<span class="mf">0.5</span> <span class="o">*</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="mf">4.0</span> <span class="o">*</span> <span class="n">direction</span><span class="p">,</span> <span class="o">-</span><span class="mi">8</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">+=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span></code></pre></figure>
<p>The specific values are determined by trial and error. You can change them
around if you have other preferences. Note that we don’t set the player
position directly, instead we modify the velocity vector and add that to the
position. This is important for collision detection.</p>
<video controls="" muted="" poster="/public/platformer/video-preview1.png">
<source src="/public/platformer/platformer-moving.mp4" type="video/mp4" />
</video>
<p>While I tried to move around as if the walls and ground had an effect, I
probably failed at deceiving you and you noticed that we still don’t have any
collision detection and handling. This code is largely adapted from Teeworlds.
It works by checking for horizontal and vertical collisions with a tile in
<code class="language-plaintext highlighter-rouge">moveBox</code>, which manipulates the player’s position based on the passed velocity vector.</p>
<p>When a collision occurs the player is moved just out of the tile in the right
direction. For the sake of simplicity in <code class="language-plaintext highlighter-rouge">isSolid</code> we consider every tile other
than <code class="language-plaintext highlighter-rouge">air, start, finish</code> a solid block. Floating point player positions are
converted to indices in the tile map.</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">math</span>
<span class="k">type</span>
<span class="n">Collision</span> <span class="p">{.</span><span class="n">pure</span><span class="p">.}</span> <span class="o">=</span> <span class="k">enum</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">corner</span>
<span class="k">const</span>
<span class="n">playerSize</span> <span class="o">=</span> <span class="n">vector2d</span><span class="p">(</span><span class="mi">64</span><span class="p">,</span> <span class="mi">64</span><span class="p">)</span>
<span class="n">air</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">start</span> <span class="o">=</span> <span class="mi">78</span>
<span class="n">finish</span> <span class="o">=</span> <span class="mi">110</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">getTile</code> reads a tile from a specified position of the map, making sure not to
over- or underflow:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">getTile</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="n">uint8</span> <span class="o">=</span>
<span class="k">let</span>
<span class="n">nx</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">x</span> <span class="ow">div</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">map</span><span class="p">.</span><span class="n">width</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">ny</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">y</span> <span class="ow">div</span> <span class="n">tileSize</span><span class="p">.</span><span class="n">y</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">map</span><span class="p">.</span><span class="n">height</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">pos</span> <span class="o">=</span> <span class="n">ny</span> <span class="o">*</span> <span class="n">map</span><span class="p">.</span><span class="n">width</span> <span class="o">+</span> <span class="n">nx</span>
<span class="n">map</span><span class="p">.</span><span class="n">tiles</span><span class="o">[</span><span class="n">pos</span><span class="o">]</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">isSolid</code> determines whether the player can collide with a tile. As we said
every tile other than air, start and finish are able to be collided with:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">isSolid</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="kt">bool</span> <span class="o">=</span>
<span class="n">map</span><span class="p">.</span><span class="n">getTile</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span> <span class="ow">notin</span> <span class="p">{</span><span class="n">air</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">finish</span><span class="p">}</span>
<span class="k">proc </span><span class="nf">isSolid</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">point</span><span class="p">:</span> <span class="n">Point2d</span><span class="p">):</span> <span class="kt">bool</span> <span class="o">=</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point</span><span class="p">.</span><span class="n">x</span><span class="p">.</span><span class="n">round</span><span class="p">.</span><span class="kt">int</span><span class="p">,</span> <span class="n">point</span><span class="p">.</span><span class="n">y</span><span class="p">.</span><span class="n">round</span><span class="p">.</span><span class="kt">int</span><span class="p">)</span></code></pre></figure>
<p>A player is on the ground when there is a block below either side of its feet:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">onGround</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">pos</span><span class="p">:</span> <span class="n">Point2d</span><span class="p">,</span> <span class="n">size</span><span class="p">:</span> <span class="n">Vector2d</span><span class="p">):</span> <span class="kt">bool</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">size</span> <span class="o">=</span> <span class="n">size</span> <span class="o">*</span> <span class="mf">0.5</span>
<span class="n">result</span> <span class="o">=</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">size</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span> <span class="ow">or</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span></code></pre></figure>
<p>Meanwhile <code class="language-plaintext highlighter-rouge">testBox</code> considers the player as an axis aligned boundary box, and
tells us if the player is stuck inside of any solid walls:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">testBox</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">pos</span><span class="p">:</span> <span class="n">Point2d</span><span class="p">,</span> <span class="n">size</span><span class="p">:</span> <span class="n">Vector2d</span><span class="p">):</span> <span class="kt">bool</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">size</span> <span class="o">=</span> <span class="n">size</span> <span class="o">*</span> <span class="mf">0.5</span>
<span class="n">result</span> <span class="o">=</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">size</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">size</span><span class="p">.</span><span class="n">y</span><span class="p">))</span> <span class="ow">or</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">size</span><span class="p">.</span><span class="n">y</span><span class="p">))</span> <span class="ow">or</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">size</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">y</span><span class="p">))</span> <span class="ow">or</span>
<span class="n">map</span><span class="p">.</span><span class="n">isSolid</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span> <span class="o">+</span> <span class="n">size</span><span class="p">.</span><span class="n">y</span><span class="p">))</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">moveBox</code> now tries to apply the velocity vector <code class="language-plaintext highlighter-rouge">vel</code> to the player’s position
<code class="language-plaintext highlighter-rouge">pos</code>. When this causes a collision the code tries to move the player only
along the x axis, then only along the y axis, to find out which side of the
tile the player collided with. If the player did not collide with any of the
sides it hit a corner (that’s the real corner case!):</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">moveBox</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">pos</span><span class="p">:</span> <span class="k">var</span> <span class="n">Point2d</span><span class="p">,</span> <span class="n">vel</span><span class="p">:</span> <span class="k">var</span> <span class="n">Vector2d</span><span class="p">,</span>
<span class="n">size</span><span class="p">:</span> <span class="n">Vector2d</span><span class="p">):</span> <span class="kt">set</span><span class="o">[</span><span class="n">Collision</span><span class="o">]</span> <span class="p">{.</span><span class="n">discardable</span><span class="p">.}</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">vel</span><span class="p">.</span><span class="n">len</span>
<span class="k">let</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">distance</span><span class="p">.</span><span class="kt">int</span>
<span class="k">if</span> <span class="n">distance</span> <span class="o"><</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">let</span> <span class="n">fraction</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="kt">float</span><span class="p">(</span><span class="n">maximum</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">0</span> <span class="p">..</span> <span class="n">maximum</span><span class="p">:</span>
<span class="k">var</span> <span class="n">newPos</span> <span class="o">=</span> <span class="n">pos</span> <span class="o">+</span> <span class="n">vel</span> <span class="o">*</span> <span class="n">fraction</span>
<span class="k">if</span> <span class="n">map</span><span class="p">.</span><span class="n">testBox</span><span class="p">(</span><span class="n">newPos</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span>
<span class="k">var</span> <span class="n">hit</span> <span class="o">=</span> <span class="kp">false</span>
<span class="k">if</span> <span class="n">map</span><span class="p">.</span><span class="n">testBox</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">newPos</span><span class="p">.</span><span class="n">y</span><span class="p">),</span> <span class="n">size</span><span class="p">):</span>
<span class="n">result</span><span class="p">.</span><span class="n">incl</span> <span class="n">Collision</span><span class="p">.</span><span class="n">y</span>
<span class="n">newPos</span><span class="p">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span>
<span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">hit</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">if</span> <span class="n">map</span><span class="p">.</span><span class="n">testBox</span><span class="p">(</span><span class="n">point2d</span><span class="p">(</span><span class="n">newPos</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span><span class="p">),</span> <span class="n">size</span><span class="p">):</span>
<span class="n">result</span><span class="p">.</span><span class="n">incl</span> <span class="n">Collision</span><span class="p">.</span><span class="n">x</span>
<span class="n">newPos</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">pos</span><span class="p">.</span><span class="n">x</span>
<span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">hit</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">hit</span><span class="p">:</span>
<span class="n">result</span><span class="p">.</span><span class="n">incl</span> <span class="n">Collision</span><span class="p">.</span><span class="n">corner</span>
<span class="n">newPos</span> <span class="o">=</span> <span class="n">pos</span>
<span class="n">vel</span> <span class="o">=</span> <span class="n">vector2d</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">pos</span> <span class="o">=</span> <span class="n">newPos</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">moveBox</code> proc also returns a set of collisions, telling us what kind of
collisions happened in this iteration. We don’t use that information, but it
could be useful if you want to handle collisions in a special way instead of
just pushing the player out of the collided wall.</p>
<p>Finally we can use <code class="language-plaintext highlighter-rouge">onGround</code> and <code class="language-plaintext highlighter-rouge">moveBox</code>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">physics</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">restart</span><span class="o">]</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">restartPlayer</span><span class="p">()</span>
<span class="k">let</span> <span class="n">ground</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">map</span><span class="p">.</span><span class="n">onGround</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">,</span> <span class="n">playerSize</span><span class="p">)</span>
<span class="k">if</span> <span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">jump</span><span class="o">]</span><span class="p">:</span>
<span class="k">if</span> <span class="n">ground</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">=</span> <span class="o">-</span><span class="mi">21</span>
<span class="k">let</span> <span class="n">direction</span> <span class="o">=</span> <span class="kt">float</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">right</span><span class="o">]</span><span class="p">.</span><span class="kt">int</span> <span class="o">-</span>
<span class="n">game</span><span class="p">.</span><span class="n">inputs</span><span class="o">[</span><span class="n">Input</span><span class="p">.</span><span class="n">left</span><span class="o">]</span><span class="p">.</span><span class="kt">int</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">y</span> <span class="o">+=</span> <span class="mf">0.75</span>
<span class="k">if</span> <span class="n">ground</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="mf">4.0</span> <span class="o">*</span> <span class="n">direction</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mf">0.95</span> <span class="o">*</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="mf">2.0</span> <span class="o">*</span> <span class="n">direction</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="o">-</span><span class="mi">8</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">map</span><span class="p">.</span><span class="n">moveBox</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">,</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">vel</span><span class="p">,</span> <span class="n">playerSize</span><span class="p">)</span></code></pre></figure>
<p>Jumping is only possible when standing on the ground. Horizontal movement in
the air is calculated in a different way than on the ground, simulating
different air and ground friction.</p>
<video controls="" muted="" poster="/public/platformer/video-preview1.png">
<source src="/public/platformer/platformer-collisions.mp4" type="video/mp4" />
</video>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part5.nim">Full code for section 5</a></p>
<h2 id="6-camera">6. Camera</h2>
<p>Did you see me disappear out of the window? Nice, right?! Well, actually that
probably makes playing the rest of the map rather difficult. So while we can
move nicely, the camera’s position is still fixed. In this section we will only
make the camera move horizontally. With this information you can probably
figure out how to make it move vertically as well if you want to. We only need
to set the camera position when our player moved, right after calling
<code class="language-plaintext highlighter-rouge">game.physics()</code>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">const</span>
<span class="n">windowSize</span><span class="p">:</span> <span class="n">Point</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1280.</span><span class="n">cint</span><span class="p">,</span> <span class="mf">720.</span><span class="n">cint</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">moveCamera</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">halfWin</span> <span class="o">=</span> <span class="kt">float</span><span class="p">(</span><span class="n">windowSize</span><span class="p">.</span><span class="n">x</span> <span class="ow">div</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">halfWin</span>
<span class="k">for</span> <span class="n">tick</span> <span class="ow">in</span> <span class="n">lastTick</span><span class="o">+</span><span class="mi">1</span> <span class="p">..</span> <span class="n">newTick</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">physics</span><span class="p">()</span>
<span class="n">game</span><span class="p">.</span><span class="n">moveCamera</span><span class="p">()</span></code></pre></figure>
<video controls="" muted="" poster="/public/platformer/video-preview3.png">
<source src="/public/platformer/platformer-camera1.mp4" type="video/mp4" />
</video>
<p>Now the camera always immediately follows the player’s horizontal position.
Another approach for the camera is to have it follow the player only once the
player leaves the center area of the screen, in this case 200 pixels:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">moveCamera</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">halfWin</span> <span class="o">=</span> <span class="kt">float</span><span class="p">(</span><span class="n">windowSize</span><span class="p">.</span><span class="n">x</span> <span class="ow">div</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">let</span>
<span class="n">leftArea</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">halfWin</span> <span class="o">-</span> <span class="mi">100</span>
<span class="n">rightArea</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">halfWin</span> <span class="o">+</span> <span class="mi">100</span>
<span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">leftArea</span><span class="p">,</span> <span class="n">rightArea</span><span class="p">)</span></code></pre></figure>
<video controls="" muted="" poster="/public/platformer/video-preview2.png">
<source src="/public/platformer/platformer-camera2.mp4" type="video/mp4" />
</video>
<p>An alternative approach is to have the camera follow the player fluidly. When
the player is further away from the center the camera moves towards him faster.
You can imagine the camera as being pulled by a rubber band connecting it to
the player, the further away they get, the stronger the camera gets pulled to
the player:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">moveCamera</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">halfWin</span> <span class="o">=</span> <span class="kt">float</span><span class="p">(</span><span class="n">windowSize</span><span class="p">.</span><span class="n">x</span> <span class="ow">div</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">let</span> <span class="n">dist</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="n">halfWin</span>
<span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">-=</span> <span class="mf">0.05</span> <span class="o">*</span> <span class="n">dist</span></code></pre></figure>
<video controls="" muted="" poster="/public/platformer/video-preview3.png">
<source src="/public/platformer/platformer-camera3.mp4" type="video/mp4" />
</video>
<p>Choices are difficult so we can just implement all three and you can choose at
compile time using <code class="language-plaintext highlighter-rouge">-d:fluidCamera</code> and <code class="language-plaintext highlighter-rouge">-d:innerCamera</code>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">moveCamera</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">halfWin</span> <span class="o">=</span> <span class="kt">float</span><span class="p">(</span><span class="n">windowSize</span><span class="p">.</span><span class="n">x</span> <span class="ow">div</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">when</span> <span class="n">defined</span><span class="p">(</span><span class="n">fluidCamera</span><span class="p">):</span>
<span class="k">let</span> <span class="n">dist</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">+</span> <span class="n">halfWin</span>
<span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">-=</span> <span class="mf">0.05</span> <span class="o">*</span> <span class="n">dist</span>
<span class="k">elif</span> <span class="n">defined</span><span class="p">(</span><span class="n">innerCamera</span><span class="p">):</span>
<span class="k">let</span>
<span class="n">leftArea</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">halfWin</span> <span class="o">-</span> <span class="mi">100</span>
<span class="n">rightArea</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">halfWin</span> <span class="o">+</span> <span class="mi">100</span>
<span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">leftArea</span><span class="p">,</span> <span class="n">rightArea</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">halfWin</span></code></pre></figure>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part6.nim">Full code for section 6</a></p>
<h2 id="7-game-state">7. Game State</h2>
<p>Now that we can run around all of the map with the camera keeping up with us,
we should give our game a purpose. You might have noticed the light and dark
gray lines at the beginning and end of the map, suspiciously referred to as
<code class="language-plaintext highlighter-rouge">start</code> and <code class="language-plaintext highlighter-rouge">finish</code> respectively. As you can probably guess we will use those
as start and finish lines and record how quickly the player can get from start
to finish:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">type</span>
<span class="n">Time</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">begin</span><span class="p">,</span> <span class="n">finish</span><span class="p">,</span> <span class="n">best</span><span class="p">:</span> <span class="kt">int</span>
<span class="n">Player</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span>
<span class="n">pos</span><span class="p">:</span> <span class="n">Point2d</span>
<span class="n">vel</span><span class="p">:</span> <span class="n">Vector2d</span>
<span class="n">time</span><span class="p">:</span> <span class="n">Time</span>
<span class="k">proc </span><span class="nf">restartPlayer</span><span class="p">(</span><span class="n">player</span><span class="p">:</span> <span class="n">Player</span><span class="p">)</span> <span class="o">=</span>
<span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">=</span> <span class="n">point2d</span><span class="p">(</span><span class="mi">170</span><span class="p">,</span> <span class="mi">500</span><span class="p">)</span>
<span class="n">player</span><span class="p">.</span><span class="n">vel</span> <span class="o">=</span> <span class="n">vector2d</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">player</span><span class="p">.</span><span class="n">time</span><span class="p">.</span><span class="n">begin</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="n">player</span><span class="p">.</span><span class="n">time</span><span class="p">.</span><span class="n">finish</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="k">proc </span><span class="nf">newTime</span><span class="p">:</span> <span class="n">Time</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">finish</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="n">result</span><span class="p">.</span><span class="n">best</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="k">proc </span><span class="nf">newPlayer</span><span class="p">(</span><span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span><span class="p">):</span> <span class="n">Player</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">texture</span> <span class="o">=</span> <span class="n">texture</span>
<span class="n">result</span><span class="p">.</span><span class="n">time</span> <span class="o">=</span> <span class="n">newTime</span><span class="p">()</span>
<span class="n">result</span><span class="p">.</span><span class="n">restartPlayer</span><span class="p">()</span></code></pre></figure>
<p>We’re now storing a <code class="language-plaintext highlighter-rouge">Time</code> object in the <code class="language-plaintext highlighter-rouge">Player</code>, telling us when the player
began playing this round, how he finished last time and what his absolute best
time is. By default the values are initialized to <code class="language-plaintext highlighter-rouge">-1</code> to indicate an invalid
value, otherwise they store ticks.</p>
<p>To format the time for display we use the excellent
<a href="http://lyro.bitbucket.org/strfmt/">strfmt</a> library’s string interpolation:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">strfmt</span>
<span class="k">proc </span><span class="nf">formatTime</span><span class="p">(</span><span class="n">ticks</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span>
<span class="k">let</span>
<span class="n">mins</span> <span class="o">=</span> <span class="p">(</span><span class="n">ticks</span> <span class="ow">div</span> <span class="mi">50</span><span class="p">)</span> <span class="ow">div</span> <span class="mi">60</span>
<span class="n">secs</span> <span class="o">=</span> <span class="p">(</span><span class="n">ticks</span> <span class="ow">div</span> <span class="mi">50</span><span class="p">)</span> <span class="ow">mod</span> <span class="mi">60</span>
<span class="n">cents</span> <span class="o">=</span> <span class="p">(</span><span class="n">ticks</span> <span class="ow">mod</span> <span class="mi">50</span><span class="p">)</span> <span class="o">*</span> <span class="mi">2</span>
<span class="s">interp"${mins:02}:${secs:02}:${cents:02}"</span></code></pre></figure>
<p>Our game logic works as follows:</p>
<ul>
<li>When the player walks through a start tile his time begins</li>
<li>When the player walks through a finish tile his finish time is set and
printed to the terminal. If it is a new best time, so is his best time. The
start time is reset.</li>
</ul>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">getTile</span><span class="p">(</span><span class="n">map</span><span class="p">:</span> <span class="n">Map</span><span class="p">,</span> <span class="n">pos</span><span class="p">:</span> <span class="n">Point2d</span><span class="p">):</span> <span class="n">uint8</span> <span class="o">=</span>
<span class="n">map</span><span class="p">.</span><span class="n">getTile</span><span class="p">(</span><span class="n">pos</span><span class="p">.</span><span class="n">x</span><span class="p">.</span><span class="n">round</span><span class="p">.</span><span class="kt">int</span><span class="p">,</span> <span class="n">pos</span><span class="p">.</span><span class="n">y</span><span class="p">.</span><span class="n">round</span><span class="p">.</span><span class="kt">int</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">logic</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">tick</span><span class="p">:</span> <span class="kt">int</span><span class="p">)</span> <span class="o">=</span>
<span class="k">template</span> <span class="n">time</span><span class="p">:</span> <span class="n">untyped</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">time</span>
<span class="k">case</span> <span class="n">game</span><span class="p">.</span><span class="n">map</span><span class="p">.</span><span class="n">getTile</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span><span class="p">)</span>
<span class="k">of</span> <span class="n">start</span><span class="p">:</span>
<span class="n">time</span><span class="p">.</span><span class="n">begin</span> <span class="o">=</span> <span class="n">tick</span>
<span class="k">of</span> <span class="n">finish</span><span class="p">:</span>
<span class="k">if</span> <span class="n">time</span><span class="p">.</span><span class="n">begin</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">time</span><span class="p">.</span><span class="n">finish</span> <span class="o">=</span> <span class="n">tick</span> <span class="o">-</span> <span class="n">time</span><span class="p">.</span><span class="n">begin</span>
<span class="n">time</span><span class="p">.</span><span class="n">begin</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="k">if</span> <span class="n">time</span><span class="p">.</span><span class="n">best</span> <span class="o"><</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">time</span><span class="p">.</span><span class="n">finish</span> <span class="o"><</span> <span class="n">time</span><span class="p">.</span><span class="n">best</span><span class="p">:</span>
<span class="n">time</span><span class="p">.</span><span class="n">best</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="n">finish</span>
<span class="n">echo</span> <span class="s">"Finished in "</span><span class="p">,</span> <span class="n">formatTime</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">finish</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span> <span class="k">discard</span></code></pre></figure>
<p>We need to call the <code class="language-plaintext highlighter-rouge">logic</code> in our <code class="language-plaintext highlighter-rouge">main</code> proc of course:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">for</span> <span class="n">tick</span> <span class="ow">in</span> <span class="n">lastTick</span><span class="o">+</span><span class="mi">1</span> <span class="p">..</span> <span class="n">newTick</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">physics</span><span class="p">()</span>
<span class="n">game</span><span class="p">.</span><span class="n">moveCamera</span><span class="p">()</span>
<span class="n">game</span><span class="p">.</span><span class="n">logic</span><span class="p">(</span><span class="n">tick</span><span class="p">)</span></code></pre></figure>
<p>Now we can play through the map and finally get an output like this on the
terminal:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Finished in 00:04:38
</code></pre></div></div>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part7.nim">Full code for section 7</a></p>
<h2 id="8-text-rendering">8. Text Rendering</h2>
<p>It would be much nicer to have the text output with the result in the actual
game window instead of on the terminal. For this we will now use SDL_ttf:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">sdl2</span><span class="o">/</span><span class="n">ttf</span>
<span class="k">proc </span><span class="nf">renderText</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">,</span> <span class="n">font</span><span class="p">:</span> <span class="n">FontPtr</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">)</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">surface</span> <span class="o">=</span> <span class="n">font</span><span class="p">.</span><span class="n">renderUtf8Solid</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="n">cstring</span><span class="p">,</span> <span class="n">color</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">surface</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Could not render text surface"</span>
<span class="k">discard</span> <span class="n">surface</span><span class="p">.</span><span class="n">setSurfaceAlphaMod</span><span class="p">(</span><span class="n">color</span><span class="p">.</span><span class="n">a</span><span class="p">)</span>
<span class="k">var</span> <span class="n">source</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">h</span><span class="p">)</span>
<span class="k">var</span> <span class="n">dest</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">h</span><span class="p">)</span>
<span class="k">let</span> <span class="n">texture</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">.</span><span class="n">createTextureFromSurface</span><span class="p">(</span><span class="n">surface</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">texture</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span>
<span class="s">"Could not create texture from rendered text"</span>
<span class="n">surface</span><span class="p">.</span><span class="n">freeSurface</span><span class="p">()</span>
<span class="n">renderer</span><span class="p">.</span><span class="n">copyEx</span><span class="p">(</span><span class="n">texture</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">dest</span><span class="p">,</span> <span class="n">angle</span> <span class="o">=</span> <span class="mf">0.0</span><span class="p">,</span> <span class="n">center</span> <span class="o">=</span> <span class="k">nil</span><span class="p">,</span>
<span class="n">flip</span> <span class="o">=</span> <span class="n">SDL_FLIP_NONE</span><span class="p">)</span>
<span class="n">texture</span><span class="p">.</span><span class="n">destroy</span><span class="p">()</span>
<span class="k">proc </span><span class="nf">renderText</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">outlineColor</span> <span class="o">=</span> <span class="n">color</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">64</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">font</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">color</span><span class="p">)</span></code></pre></figure>
<p>What’s happening here is that we render a text with a <code class="language-plaintext highlighter-rouge">FontPtr</code> to an SDL2
surface (stored in RAM), which is then put into a texture (stored in GPU’s
VRAM). This texture is then rendered to the screen at the defined position.</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">render</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">tick</span><span class="p">:</span> <span class="kt">int</span><span class="p">)</span> <span class="o">=</span>
<span class="c"># Draw over all drawings of the last frame with the default color</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="c"># Actual drawing here</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderTee</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">texture</span><span class="p">,</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">-</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderMap</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">map</span><span class="p">,</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="k">let</span> <span class="n">time</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">time</span>
<span class="k">const</span> <span class="n">white</span> <span class="o">=</span> <span class="n">color</span><span class="p">(</span><span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">)</span>
<span class="k">if</span> <span class="n">time</span><span class="p">.</span><span class="n">begin</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="n">formatTime</span><span class="p">(</span><span class="n">tick</span> <span class="o">-</span> <span class="n">time</span><span class="p">.</span><span class="n">begin</span><span class="p">),</span> <span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="n">white</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">time</span><span class="p">.</span><span class="n">finish</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="s">"Finished in: "</span> <span class="o">&</span> <span class="n">formatTime</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">finish</span><span class="p">),</span>
<span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="n">white</span><span class="p">)</span>
<span class="k">if</span> <span class="n">time</span><span class="p">.</span><span class="n">best</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="s">"Best time: "</span> <span class="o">&</span> <span class="n">formatTime</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">best</span><span class="p">),</span>
<span class="mi">50</span><span class="p">,</span> <span class="mi">150</span><span class="p">,</span> <span class="n">white</span><span class="p">)</span>
<span class="c"># Show the result on screen</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">present</span><span class="p">()</span></code></pre></figure>
<p>We need to initialize the TTF subsystem and now we also need to pass the
current tick to <code class="language-plaintext highlighter-rouge">render</code> inside of our <code class="language-plaintext highlighter-rouge">main</code> function:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="n">sdlFailIf</span><span class="p">(</span><span class="n">ttfInit</span><span class="p">()</span> <span class="o">==</span> <span class="n">SdlError</span><span class="p">):</span> <span class="s">"SDL2 TTF initialization failed"</span>
<span class="k">defer</span><span class="p">:</span> <span class="n">ttfQuit</span><span class="p">()</span>
<span class="p">...</span>
<span class="n">game</span><span class="p">.</span><span class="n">render</span><span class="p">(</span><span class="n">lastTick</span><span class="p">)</span></code></pre></figure>
<p>As well as loading our <a href="https://github.com/def-/nim-platformer/blob/master/tutorial/DejaVuSans.ttf?raw=true">DejaVuSans.ttf</a> font inside of <code class="language-plaintext highlighter-rouge">newGame</code>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="n">result</span><span class="p">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">openFont</span><span class="p">(</span><span class="s">"DejaVuSans.ttf"</span><span class="p">,</span> <span class="mi">28</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">result</span><span class="p">.</span><span class="n">font</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Failed to load font"</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">Game</code> object now looks like this, including a <code class="language-plaintext highlighter-rouge">font</code>:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">type</span>
<span class="n">Game</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">inputs</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="n">Input</span><span class="p">,</span> <span class="kt">bool</span><span class="o">]</span>
<span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span>
<span class="n">font</span><span class="p">:</span> <span class="n">FontPtr</span>
<span class="n">player</span><span class="p">:</span> <span class="n">Player</span>
<span class="n">map</span><span class="p">:</span> <span class="n">Map</span>
<span class="n">camera</span><span class="p">:</span> <span class="n">Vector2d</span></code></pre></figure>
<p>This is what the text rendering looks like after finishing the map:</p>
<p><img src="/public/platformer/platformer-time1.png" alt="Finished time 1" /></p>
<p>It noticeably doesn’t look all that great. The border of the text looks rugged.
This happens because <code class="language-plaintext highlighter-rouge">renderUtf8Solid</code> does not use alpha blending, which is
expensive. Instead every pixel is either entirely white or entirely
transparent, never anything half-transparent in between. If we had a fixed
background color for the text we could use <code class="language-plaintext highlighter-rouge">renderUtf8Shaded</code>, which takes a
background color. If we want nicer output with dynamic backgrounds we can use
<code class="language-plaintext highlighter-rouge">renderUtf8Blended</code> instead:</p>
<p><img src="/public/platformer/platformer-time2.png" alt="Finished time 2" /></p>
<p>This looks better, but would be hard to see with a brighter background. We
can draw an outline for our text to fix this, basically by drawing the text
twice, once in half-transparent black and once in the proper color on top:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">renderText</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">,</span> <span class="n">font</span><span class="p">:</span> <span class="n">FontPtr</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">outline</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">)</span> <span class="o">=</span>
<span class="n">font</span><span class="p">.</span><span class="n">setFontOutline</span><span class="p">(</span><span class="n">outline</span><span class="p">)</span>
<span class="k">let</span> <span class="n">surface</span> <span class="o">=</span> <span class="n">font</span><span class="p">.</span><span class="n">renderUtf8Blended</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="n">cstring</span><span class="p">,</span> <span class="n">color</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">surface</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Could not render text surface"</span>
<span class="k">discard</span> <span class="n">surface</span><span class="p">.</span><span class="n">setSurfaceAlphaMod</span><span class="p">(</span><span class="n">color</span><span class="p">.</span><span class="n">a</span><span class="p">)</span>
<span class="k">var</span> <span class="n">source</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">h</span><span class="p">)</span>
<span class="k">var</span> <span class="n">dest</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">outline</span><span class="p">,</span> <span class="n">y</span> <span class="o">-</span> <span class="n">outline</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">surface</span><span class="p">.</span><span class="n">h</span><span class="p">)</span>
<span class="p">...</span>
<span class="k">proc </span><span class="nf">renderText</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">outlineColor</span> <span class="o">=</span> <span class="n">color</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">64</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">font</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">outlineColor</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">font</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">color</span><span class="p">)</span></code></pre></figure>
<p><img src="/public/platformer/platformer-time3.png" alt="Finished time 3" /></p>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part8.nim">Full code for section 8</a></p>
<h2 id="9-text-caching">9. Text Caching</h2>
<p>If you looked at your CPU usage during this tutorial so far you might have
noticed that the game needed nearly no CPU at all, about 3 % for 60 fps on my
system. Once the texts are rendered this increases to 20% though. The reason
for this is that right now we are regenerating the text textures every single
frame, even if it didn’t change from the last frame. If you have fixed texts
you can simply save the textures instead of recalculating them. But if you want
more flexibility instead you can use a glyph or texture caching system. An
example of such a system would be <a href="https://github.com/grimfang4/SDL_FontCache">SDL_FontCache</a>.</p>
<p>But instead we can write a little application-specific caching scheme in Nim.
The heuristic we use is that mostly a single line in the code base will keep
producing the same string, at least for some time. So we cache only a single
text rendering for each line that prints something to the screen. That means we
don’t have to do any lookups in a cache data structure and we only use a
guaranteed constant amount of memory for caching:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">type</span>
<span class="n">CacheLine</span> <span class="o">=</span> <span class="k">object</span>
<span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span>
<span class="n">w</span><span class="p">,</span> <span class="n">h</span><span class="p">:</span> <span class="n">cint</span>
<span class="n">TextCache</span> <span class="o">=</span> <span class="k">ref</span> <span class="k">object</span>
<span class="n">text</span><span class="p">:</span> <span class="kt">string</span>
<span class="n">cache</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="mi">2</span><span class="p">,</span> <span class="n">CacheLine</span><span class="o">]</span>
<span class="k">proc </span><span class="nf">newTextCache</span><span class="p">:</span> <span class="n">TextCache</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span></code></pre></figure>
<p>A <code class="language-plaintext highlighter-rouge">CacheLine</code> is what we store, a pointer to a texture as well as the texture’s
width and height. For our text rendering we need two of those as we render the
text twice to get the outline effect. The <code class="language-plaintext highlighter-rouge">text</code> is also stored in <code class="language-plaintext highlighter-rouge">TextCache</code>
to see if we already have the correct textures cached.</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">renderText</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">,</span> <span class="n">font</span><span class="p">:</span> <span class="n">FontPtr</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">outline</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">):</span> <span class="n">CacheLine</span> <span class="o">=</span>
<span class="n">font</span><span class="p">.</span><span class="n">setFontOutline</span><span class="p">(</span><span class="n">outline</span><span class="p">)</span>
<span class="k">let</span> <span class="n">surface</span> <span class="o">=</span> <span class="n">font</span><span class="p">.</span><span class="n">renderUtf8Blended</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="n">cstring</span><span class="p">,</span> <span class="n">color</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">surface</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Could not render text surface"</span>
<span class="k">discard</span> <span class="n">surface</span><span class="p">.</span><span class="n">setSurfaceAlphaMod</span><span class="p">(</span><span class="n">color</span><span class="p">.</span><span class="n">a</span><span class="p">)</span>
<span class="n">result</span><span class="p">.</span><span class="n">w</span> <span class="o">=</span> <span class="n">surface</span><span class="p">.</span><span class="n">w</span>
<span class="n">result</span><span class="p">.</span><span class="n">h</span> <span class="o">=</span> <span class="n">surface</span><span class="p">.</span><span class="n">h</span>
<span class="n">result</span><span class="p">.</span><span class="n">texture</span> <span class="o">=</span> <span class="n">renderer</span><span class="p">.</span><span class="n">createTextureFromSurface</span><span class="p">(</span><span class="n">surface</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">result</span><span class="p">.</span><span class="n">texture</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Could not create texture from rendered text"</span>
<span class="n">surface</span><span class="p">.</span><span class="n">freeSurface</span><span class="p">()</span></code></pre></figure>
<p>We manipulated <code class="language-plaintext highlighter-rouge">renderText</code> to return a <code class="language-plaintext highlighter-rouge">CacheLine</code> that we can use:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">renderText</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span>
<span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">,</span> <span class="n">tc</span><span class="p">:</span> <span class="n">TextCache</span><span class="p">)</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">passes</span> <span class="o">=</span> <span class="o">[</span><span class="p">(</span><span class="n">color</span><span class="p">:</span> <span class="n">color</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">64</span><span class="p">),</span> <span class="n">outline</span><span class="p">:</span> <span class="mf">2.</span><span class="n">cint</span><span class="p">),</span>
<span class="p">(</span><span class="n">color</span><span class="p">:</span> <span class="n">color</span><span class="p">,</span> <span class="n">outline</span><span class="p">:</span> <span class="mf">0.</span><span class="n">cint</span><span class="p">)</span><span class="o">]</span>
<span class="k">if</span> <span class="n">text</span> <span class="o">!=</span> <span class="n">tc</span><span class="p">.</span><span class="n">text</span><span class="p">:</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">0</span><span class="p">..</span><span class="mi">1</span><span class="p">:</span>
<span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">texture</span><span class="p">.</span><span class="n">destroy</span><span class="p">()</span>
<span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span>
<span class="n">game</span><span class="p">.</span><span class="n">font</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">passes</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">outline</span><span class="p">,</span> <span class="n">passes</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">color</span><span class="p">)</span>
<span class="n">tc</span><span class="p">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">text</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">0</span><span class="p">..</span><span class="mi">1</span><span class="p">:</span>
<span class="k">var</span> <span class="n">source</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">h</span><span class="p">)</span>
<span class="k">var</span> <span class="n">dest</span> <span class="o">=</span> <span class="n">rect</span><span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">passes</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">outline</span><span class="p">,</span> <span class="n">y</span> <span class="o">-</span> <span class="n">passes</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">outline</span><span class="p">,</span>
<span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">h</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">copyEx</span><span class="p">(</span><span class="n">tc</span><span class="p">.</span><span class="n">cache</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">texture</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">dest</span><span class="p">,</span>
<span class="n">angle</span> <span class="o">=</span> <span class="mf">0.0</span><span class="p">,</span> <span class="n">center</span> <span class="o">=</span> <span class="k">nil</span><span class="p">)</span></code></pre></figure>
<p>In two passes the text is rendered, if the text is cached from the cache
directly, otherwise the old cache entry is removed and replaced with the new
textures.</p>
<p>Nim’s metaprogramming allows us to use this seamlessly with a small template. A
few days ago I wrote an article about <a href="https://hookrace.net/blog/introduction-to-metaprogramming-in-nim/">Metaprogramming in
Nim</a> if you
want to learn more about the powerful side of Nim.</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">template</span> <span class="n">renderTextCached</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">color</span><span class="p">:</span> <span class="n">Color</span><span class="p">)</span> <span class="o">=</span>
<span class="k">block</span><span class="p">:</span>
<span class="k">var</span> <span class="n">tc</span> <span class="p">{.</span><span class="n">global</span><span class="p">.}</span> <span class="o">=</span> <span class="n">newTextCache</span><span class="p">()</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderText</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">color</span><span class="p">,</span> <span class="n">tc</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">render</span><span class="p">(</span><span class="n">game</span><span class="p">:</span> <span class="n">Game</span><span class="p">,</span> <span class="n">tick</span><span class="p">:</span> <span class="kt">int</span><span class="p">)</span> <span class="o">=</span>
<span class="c"># Draw over all drawings of the last frame with the default color</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="c"># Actual drawing here</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderTee</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">texture</span><span class="p">,</span>
<span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">pos</span> <span class="o">-</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">renderMap</span><span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">map</span><span class="p">,</span> <span class="n">game</span><span class="p">.</span><span class="n">camera</span><span class="p">)</span>
<span class="k">let</span> <span class="n">time</span> <span class="o">=</span> <span class="n">game</span><span class="p">.</span><span class="n">player</span><span class="p">.</span><span class="n">time</span>
<span class="k">const</span> <span class="n">white</span> <span class="o">=</span> <span class="n">color</span><span class="p">(</span><span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">)</span>
<span class="k">if</span> <span class="n">time</span><span class="p">.</span><span class="n">begin</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderTextCached</span><span class="p">(</span><span class="n">formatTime</span><span class="p">(</span><span class="n">tick</span> <span class="o">-</span> <span class="n">time</span><span class="p">.</span><span class="n">begin</span><span class="p">),</span>
<span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="n">white</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">time</span><span class="p">.</span><span class="n">finish</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderTextCached</span><span class="p">(</span><span class="s">"Finished in: "</span> <span class="o">&</span>
<span class="n">formatTimeExact</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">finish</span><span class="p">),</span> <span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="n">white</span><span class="p">)</span>
<span class="k">if</span> <span class="n">time</span><span class="p">.</span><span class="n">best</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderTextCached</span><span class="p">(</span><span class="s">"Best time: "</span> <span class="o">&</span>
<span class="n">formatTimeExact</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">best</span><span class="p">),</span> <span class="mi">50</span><span class="p">,</span> <span class="mi">150</span><span class="p">,</span> <span class="n">white</span><span class="p">)</span>
<span class="c"># Show the result on screen</span>
<span class="n">game</span><span class="p">.</span><span class="n">renderer</span><span class="p">.</span><span class="n">present</span><span class="p">()</span></code></pre></figure>
<p>Now each of the three <code class="language-plaintext highlighter-rouge">renderTextCached</code> calls get its own <code class="language-plaintext highlighter-rouge">TextCache</code>
assigned, which is then used for the rest of the program execution. Note that
this caching scheme only works under the assumption that there are relatively
few separate lines of code that call <code class="language-plaintext highlighter-rouge">renderTextCached</code> and the ones that do
often render the same text multiple times in a row. Good enough for our use
case.</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">formatTime</span><span class="p">(</span><span class="n">ticks</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">mins</span> <span class="o">=</span> <span class="p">(</span><span class="n">ticks</span> <span class="ow">div</span> <span class="mi">50</span><span class="p">)</span> <span class="ow">div</span> <span class="mi">60</span>
<span class="k">let</span> <span class="n">secs</span> <span class="o">=</span> <span class="p">(</span><span class="n">ticks</span> <span class="ow">div</span> <span class="mi">50</span><span class="p">)</span> <span class="ow">mod</span> <span class="mi">60</span>
<span class="s">interp"${mins:02}:${secs:02}"</span>
<span class="k">proc </span><span class="nf">formatTimeExact</span><span class="p">(</span><span class="n">ticks</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">cents</span> <span class="o">=</span> <span class="p">(</span><span class="n">ticks</span> <span class="ow">mod</span> <span class="mi">50</span><span class="p">)</span> <span class="o">*</span> <span class="mi">2</span>
<span class="s">interp"${formatTime(ticks)}:${cents:02}"</span></code></pre></figure>
<p>We also reduced the exactness of the current time format because before it had
to be recalculated every single frame. Now we’re back to using 4 % CPU and our
final game looks like this:</p>
<video controls="" muted="" poster="/public/platformer/video-preview3.png">
<source src="/public/platformer/platformer-finished.mp4" type="video/mp4" />
</video>
<p><a href="https://github.com/def-/nim-platformer/blob/master/tutorial/platformer_part9.nim">Full code for section 9</a></p>
<h2 id="building">Building</h2>
<p>We can create a <code class="language-plaintext highlighter-rouge">platformer.nimble</code> file to tell
<a href="https://github.com/nim-lang/nimble">Nimble</a>, Nim’s package manager, how to
build our package:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="c"># Package</span>
<span class="n">version</span> <span class="o">=</span> <span class="s">"1.0"</span>
<span class="n">author</span> <span class="o">=</span> <span class="s">"Dennis Felsing"</span>
<span class="n">description</span> <span class="o">=</span> <span class="s">"An example platform game with SDL2"</span>
<span class="n">license</span> <span class="o">=</span> <span class="s">"MIT"</span>
<span class="n">bin</span> <span class="o">=</span> <span class="o">@[</span><span class="s">"platformer"</span><span class="o">]</span>
<span class="c"># Dependencies</span>
<span class="n">requires</span> <span class="s">"nim >= 0.10.0"</span>
<span class="n">requires</span> <span class="s">"sdl2 >= 1.1"</span>
<span class="n">requires</span> <span class="s">"strfmt >= 0.6"</span>
<span class="n">requires</span> <span class="s">"basic2d >= 0.1.0"</span>
<span class="n">task</span> <span class="n">tests</span><span class="p">,</span> <span class="s">"Compile all tutorial steps"</span><span class="p">:</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">9</span><span class="p">:</span>
<span class="n">exec</span> <span class="s">"nim c tutorial/platformer_part"</span> <span class="o">&</span> <span class="o">$</span><span class="n">i</span></code></pre></figure>
<p>All the intermediate code states from this article can be compiled with <code class="language-plaintext highlighter-rouge">nimble
tests</code>. We can test the build by running <code class="language-plaintext highlighter-rouge">nimble build</code>, which creates a
<code class="language-plaintext highlighter-rouge">platformer</code> binary that we can run. <code class="language-plaintext highlighter-rouge">nimble install</code> installs the same binary
so that it is available in <code class="language-plaintext highlighter-rouge">~/.nimble/bin</code>.</p>
<p>But once we run this installed <code class="language-plaintext highlighter-rouge">platformer</code> binary we get an error:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Error: unhandled exception: Could not load image player.png, SDL error: Couldn't open player.png [SDLException]
</code></pre></div></div>
<p>Actually this makes sense, because we load the files at runtime from the
current directory, which can now be any directory. We have two choices for how
to solve this:</p>
<ul>
<li>Read in all files at compile time and get a binary that depends on no assets<br />
Doing this is pretty simple in Nim as the compiler supports reading arbitrary files at compile time with <code class="language-plaintext highlighter-rouge">staticRead</code>:</li>
</ul>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">streams</span>
<span class="k">template</span> <span class="n">staticReadRW</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="k">ptr</span> <span class="n">RWops</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">file</span> <span class="o">=</span> <span class="n">staticRead</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">rwFromConstMem</span><span class="p">(</span><span class="n">file</span><span class="p">.</span><span class="n">cstring</span><span class="p">,</span> <span class="n">file</span><span class="p">.</span><span class="n">len</span><span class="p">)</span>
<span class="k">template</span> <span class="n">staticReadStream</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">file</span> <span class="o">=</span> <span class="n">staticRead</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">newStringStream</span><span class="p">(</span><span class="n">file</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">newGame</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">):</span> <span class="n">Game</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">renderer</span> <span class="o">=</span> <span class="n">renderer</span>
<span class="n">result</span><span class="p">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">openFontRW</span><span class="p">(</span>
<span class="n">staticReadRW</span><span class="p">(</span><span class="s">"DejaVuSans.ttf"</span><span class="p">),</span> <span class="n">freesrc</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">result</span><span class="p">.</span><span class="n">font</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Failed to load font"</span>
<span class="n">result</span><span class="p">.</span><span class="n">player</span> <span class="o">=</span> <span class="n">newPlayer</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture_RW</span><span class="p">(</span>
<span class="n">staticReadRW</span><span class="p">(</span><span class="s">"player.png"</span><span class="p">),</span> <span class="n">freesrc</span> <span class="o">=</span> <span class="mi">1</span><span class="p">))</span>
<span class="n">result</span><span class="p">.</span><span class="n">map</span> <span class="o">=</span> <span class="n">newMap</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture_RW</span><span class="p">(</span>
<span class="n">staticReadRW</span><span class="p">(</span><span class="s">"grass.png"</span><span class="p">),</span> <span class="n">freesrc</span> <span class="o">=</span> <span class="mi">1</span><span class="p">),</span>
<span class="n">staticReadStream</span><span class="p">(</span><span class="s">"default.map"</span><span class="p">))</span>
</code></pre></figure>
<ul>
<li>Define a directory where we store our data assets to load them at runtime.
This enables the player to switch them out for custom ones without the need
to recompile the binary. And as we’ve seen with the <a href="https://ddnet.org/skins/">DDNet Skin
Database</a> this might be a useful feature. We find
the data directory by looking where in the file system our binary is. Let’s
implement this and make the old embedded compile time assets optional:</li>
</ul>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">os</span><span class="p">,</span> <span class="n">streams</span>
<span class="k">const</span> <span class="n">dataDir</span> <span class="o">=</span> <span class="s">"data"</span>
<span class="k">when</span> <span class="n">defined</span><span class="p">(</span><span class="n">embedData</span><span class="p">):</span>
<span class="k">template</span> <span class="n">readRW</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="k">ptr</span> <span class="n">RWops</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">file</span> <span class="o">=</span> <span class="n">staticRead</span><span class="p">(</span><span class="n">dataDir</span> <span class="o">/</span> <span class="n">filename</span><span class="p">)</span>
<span class="n">rwFromConstMem</span><span class="p">(</span><span class="n">file</span><span class="p">.</span><span class="n">cstring</span><span class="p">,</span> <span class="n">file</span><span class="p">.</span><span class="n">len</span><span class="p">)</span>
<span class="k">template</span> <span class="n">readStream</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="n">Stream</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">file</span> <span class="o">=</span> <span class="n">staticRead</span><span class="p">(</span><span class="n">dataDir</span> <span class="o">/</span> <span class="n">filename</span><span class="p">)</span>
<span class="n">newStringStream</span><span class="p">(</span><span class="n">file</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">let</span> <span class="n">fullDataDir</span> <span class="o">=</span> <span class="n">getAppDir</span><span class="p">()</span> <span class="o">/</span> <span class="n">dataDir</span>
<span class="k">template</span> <span class="n">readRW</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="k">ptr</span> <span class="n">RWops</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">rw</span> <span class="o">=</span> <span class="n">rwFromFile</span><span class="p">(</span><span class="n">cstring</span><span class="p">(</span><span class="n">fullDataDir</span> <span class="o">/</span> <span class="n">filename</span><span class="p">),</span> <span class="s">"r"</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">rw</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Cannot create RWops from file"</span>
<span class="n">rw</span>
<span class="k">template</span> <span class="n">readStream</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="n">Stream</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">stream</span> <span class="o">=</span> <span class="n">newFileStream</span><span class="p">(</span><span class="n">fullDataDir</span> <span class="o">/</span> <span class="n">filename</span><span class="p">)</span>
<span class="k">if</span> <span class="n">stream</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="k">raise</span> <span class="n">ValueError</span><span class="p">.</span><span class="n">newException</span><span class="p">(</span>
<span class="s">"Cannot open file stream:"</span> <span class="o">&</span> <span class="n">fullDataDir</span> <span class="o">/</span> <span class="n">filename</span><span class="p">)</span>
<span class="n">stream</span>
<span class="k">proc </span><span class="nf">newGame</span><span class="p">(</span><span class="n">renderer</span><span class="p">:</span> <span class="n">RendererPtr</span><span class="p">):</span> <span class="n">Game</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">renderer</span> <span class="o">=</span> <span class="n">renderer</span>
<span class="n">result</span><span class="p">.</span><span class="n">font</span> <span class="o">=</span> <span class="n">openFontRW</span><span class="p">(</span>
<span class="n">readRW</span><span class="p">(</span><span class="s">"DejaVuSans.ttf"</span><span class="p">),</span> <span class="n">freesrc</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">)</span>
<span class="n">sdlFailIf</span> <span class="n">result</span><span class="p">.</span><span class="n">font</span><span class="p">.</span><span class="n">isNil</span><span class="p">:</span> <span class="s">"Failed to load font"</span>
<span class="n">result</span><span class="p">.</span><span class="n">player</span> <span class="o">=</span> <span class="n">newPlayer</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture_RW</span><span class="p">(</span>
<span class="n">readRW</span><span class="p">(</span><span class="s">"player.png"</span><span class="p">),</span> <span class="n">freesrc</span> <span class="o">=</span> <span class="mi">1</span><span class="p">))</span>
<span class="n">result</span><span class="p">.</span><span class="n">map</span> <span class="o">=</span> <span class="n">newMap</span><span class="p">(</span><span class="n">renderer</span><span class="p">.</span><span class="n">loadTexture_RW</span><span class="p">(</span>
<span class="n">readRW</span><span class="p">(</span><span class="s">"grass.png"</span><span class="p">),</span> <span class="n">freesrc</span> <span class="o">=</span> <span class="mi">1</span><span class="p">),</span>
<span class="n">readStream</span><span class="p">(</span><span class="s">"default.map"</span><span class="p">))</span>
</code></pre></figure>
<p>This also requires us to change <code class="language-plaintext highlighter-rouge">newMap</code> to accept a <code class="language-plaintext highlighter-rouge">Stream</code> instead of a filename:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">proc </span><span class="nf">newMap</span><span class="p">(</span><span class="n">texture</span><span class="p">:</span> <span class="n">TexturePtr</span><span class="p">,</span> <span class="n">map</span><span class="p">:</span> <span class="n">Stream</span><span class="p">):</span> <span class="n">Map</span> <span class="o">=</span>
<span class="n">new</span> <span class="n">result</span>
<span class="n">result</span><span class="p">.</span><span class="n">texture</span> <span class="o">=</span> <span class="n">texture</span>
<span class="n">result</span><span class="p">.</span><span class="n">tiles</span> <span class="o">=</span> <span class="o">@[]</span>
<span class="k">var</span> <span class="n">line</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">while</span> <span class="n">map</span><span class="p">.</span><span class="n">readLine</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
<span class="p">...</span></code></pre></figure>
<p>When compiling you can set <code class="language-plaintext highlighter-rouge">-d:embedData</code> like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim -d:release c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 129K Jun 13 14:54 platformer*
$ nim -d:release -d:embedData c platformer
$ ls -lha platformer
-rwxr-xr-x 1 deen deen 888K Jun 13 14:55 platformer*
</code></pre></div></div>
<p>You can find our <a href="https://github.com/def-/nim-platformer/blob/master/platformer.nim">final platform game
code</a> in the
<a href="https://github.com/def-/nim-platformer">repository</a>.</p>
<p>Now we can submit the repository to the <a href="https://github.com/nim-lang/packages">Nimble
packages</a> as a pull request and soon all
Nim developers can install the package right from their terminal with a <code class="language-plaintext highlighter-rouge">nimble
install platformer</code> and play by just running <code class="language-plaintext highlighter-rouge">platformer</code>.</p>
<p>A <a href="https://github.com/def-/nim-platformer/blob/master/circle.yml">circle.yml</a>
file defines how to run and compile our repository and is used to make sure it
keeps building fine whenever changes are made.</p>
<h2 id="binary-distribution">Binary Distribution</h2>
<p>But Nim developers are probably not the main target group of our game, so
ideally we we also want to be able to build binaries for a few common
platforms, Linux x86, x86-64 and Windows x86, x86-64 for us. Building for Mac
OS X is a bit more involved, but you can check out how
<a href="https://hookrace.net/blog/ddnet-evolution-architecture-technology/#software-releases">DDNet</a>
<a href="https://github.com/ddnet/ddnet-scripts/blob/master/ddnet-release.sh">does</a>
<a href="https://github.com/ddnet/ddnet/blob/master/scripts/make_release.py">it</a>.</p>
<p>Of course we could just set up a VM for each system that we want to build for
and use the instructions from the start of this article. But that’s tedious and
we want the convenience of building on a single machine.</p>
<h3 id="linux">Linux</h3>
<p>Note that I’m on Arch Linux, but this should be possible in an analogous way on
other Linux distributions:</p>
<p>Building a portable binary for Linux is a pain because of glibc. When you
compile on a system with a newer glibc version it might not run on a system
with an older one. A common solution is to use a build system with an old Linux
install. Alternatively an old Debian chroot can be created with
<a href="https://wiki.debian.org/Debootstrap">debootstrap</a>. There is also <a href="http://www.linuxfoundation.org/collaborate/workgroups/lsb">Linux
Standard Base</a> which
aims to solve this problem, but I haven’t used yet.</p>
<p>A more hacky solution is to create the binary on the new system and check what
symbols exactly are linked against the newer glibc version. In our case I want
everything to use <code class="language-plaintext highlighter-rouge">GLIBC_2.2.5</code>, so I check for anything else:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>objdump -T platformer | grep GLIBC | grep -v 2.2.5
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.14 memcpy
</code></pre></div></div>
<p>Okay, only <code class="language-plaintext highlighter-rouge">memcpy</code> is the problem. We can force the linker to use an old version
of <code class="language-plaintext highlighter-rouge">memcpy</code> and another common problem, <code class="language-plaintext highlighter-rouge">realpath</code>, like this in C code with
inline assembler:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="n">__asm__</span><span class="p">(</span><span class="s">".symver memcpy,memcpy@GLIBC_2.2.5"</span><span class="p">);</span>
<span class="n">__asm__</span><span class="p">(</span><span class="s">".symver realpath,realpath@GLIBC_2.2.5"</span><span class="p">);</span></code></pre></figure>
<p>But this would require for us to insert this into every C file that we
generate. Or we abuse the <code class="language-plaintext highlighter-rouge">nimbase.h</code> file and insert it there and compile
with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ head -n2 glibc-hack/nimbase.h
__asm__(".symver memcpy,memcpy@GLIBC_2.2.5");
__asm__(".symver realpath,realpath@GLIBC_2.2.5");
$ nim -d:release --passC:-Iglibc-hack c platformer
$ objdump -T platformer | grep memcpy
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 memcpy
$ objdump -T platformer|grep GLIBC|grep -v 2.2.5
</code></pre></div></div>
<p>Now our binary works on Linux versions using glibc 2.2.5 or newer. Note that
the user still needs SDL2, SDL_image2 and SDL_ttf2 installed.</p>
<p>When you’re linking dynamically and want to distribute shared libraries with
your binary you can compile with <code class="language-plaintext highlighter-rouge">nim --passC:-Wl,-rpath,. c platformer</code> and
put the shared libraries into the same directory as the binary.</p>
<p>At least building for x86 is easy on Linux as long as you’re on x86-64 and have
gcc-multilib installed:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ yaourt -S lib32-sdl2 lib32-sdl2_image lib32-sdl2_ttf
$ nim --cpu:i386 --passC:-m32 --passL:-m32 -d:release c platformer
</code></pre></div></div>
<h3 id="windows">Windows</h3>
<p>Surprisingly it is much easier to build portable binaries for Windows, even from Linux:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pacman -S mingw-w64-gcc
$ nim --os:windows --cpu:amd64 --gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc -d:release c platformer
$ nim --os:windows --cpu:i386 --gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc -d:release c platformer
</code></pre></div></div>
<p>SDL libraries for Windows can be downloaded from the <a href="https://www.libsdl.org/download-2.0.php">SDL2
website</a>
(<a href="https://www.libsdl.org/projects/SDL_image/">image</a>,
<a href="https://www.libsdl.org/projects/SDL_ttf/">ttf</a>).</p>
<p>Stripping the binaries of symbols with <code class="language-plaintext highlighter-rouge">strip -s platformer</code> is a good idea if
you want to save some space and don’t care about debugging your binary.</p>
<h3 id="automated-build-script">Automated Build Script</h3>
<p>With all this information we can now write a fully automated release build
script, written in Nim as well:</p>
<figure class="highlight"><pre><code class="language-nim" data-lang="nim"><span class="k">import</span> <span class="n">os</span><span class="p">,</span> <span class="n">strfmt</span>
<span class="k">const</span>
<span class="n">app</span> <span class="o">=</span> <span class="s">"platformer"</span>
<span class="n">version</span> <span class="o">=</span> <span class="s">"1.0"</span>
<span class="n">builds</span> <span class="o">=</span> <span class="o">[</span>
<span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="s">"linux_x86"</span><span class="p">,</span> <span class="n">os</span><span class="p">:</span> <span class="s">"linux"</span><span class="p">,</span> <span class="n">cpu</span><span class="p">:</span> <span class="s">"i386"</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="s">"--passC:-m32 --passL:-m32"</span><span class="p">),</span>
<span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="s">"linux_x86_64"</span><span class="p">,</span> <span class="n">os</span><span class="p">:</span> <span class="s">"linux"</span><span class="p">,</span> <span class="n">cpu</span><span class="p">:</span> <span class="s">"amd64"</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="s">"--passC:-Iglibc-hack"</span><span class="p">),</span>
<span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="s">"win32"</span><span class="p">,</span> <span class="n">os</span><span class="p">:</span> <span class="s">"windows"</span><span class="p">,</span> <span class="n">cpu</span><span class="p">:</span> <span class="s">"i386"</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="s">"--gcc.exe:i686-w64-mingw32-gcc --gcc.linkerexe:i686-w64-mingw32-gcc"</span><span class="p">),</span>
<span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="s">"win64"</span><span class="p">,</span> <span class="n">os</span><span class="p">:</span> <span class="s">"windows"</span><span class="p">,</span> <span class="n">cpu</span><span class="p">:</span> <span class="s">"amd64"</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="s">"--gcc.exe:x86_64-w64-mingw32-gcc --gcc.linkerexe:x86_64-w64-mingw32-gcc"</span><span class="p">),</span>
<span class="o">]</span>
<span class="n">removeDir</span> <span class="s">"builds"</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">os</span><span class="p">,</span> <span class="n">cpu</span><span class="p">,</span> <span class="n">args</span> <span class="ow">in</span> <span class="n">builds</span><span class="p">.</span><span class="n">items</span><span class="p">:</span>
<span class="k">let</span>
<span class="n">dirName</span> <span class="o">=</span> <span class="n">app</span> <span class="o">&</span> <span class="s">"_"</span> <span class="o">&</span> <span class="n">version</span> <span class="o">&</span> <span class="s">"_"</span> <span class="o">&</span> <span class="n">name</span>
<span class="n">dir</span> <span class="o">=</span> <span class="s">"builds"</span> <span class="o">/</span> <span class="n">dirName</span>
<span class="n">exeExt</span> <span class="o">=</span> <span class="k">if</span> <span class="n">os</span> <span class="o">==</span> <span class="s">"windows"</span><span class="p">:</span> <span class="s">".exe"</span> <span class="k">else</span><span class="p">:</span> <span class="s">""</span>
<span class="n">bin</span> <span class="o">=</span> <span class="n">dir</span> <span class="o">/</span> <span class="n">app</span> <span class="o">&</span> <span class="n">exeExt</span>
<span class="n">createDir</span> <span class="n">dir</span>
<span class="k">if</span> <span class="n">execShellCmd</span><span class="p">(</span><span class="s">interp"nim --cpu:${cpu} --os:${os} ${args} -d:release -o:${bin} c ${app}"</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="n">quit</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">execShellCmd</span><span class="p">(</span><span class="s">interp"strip -s ${bin}"</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="n">quit</span> <span class="mi">1</span>
<span class="n">copyDir</span><span class="p">(</span><span class="s">"data"</span><span class="p">,</span> <span class="n">dir</span> <span class="o">/</span> <span class="s">"data"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">os</span> <span class="o">==</span> <span class="s">"windows"</span><span class="p">:</span> <span class="n">copyDir</span><span class="p">(</span><span class="s">"libs"</span> <span class="o">/</span> <span class="n">name</span><span class="p">,</span> <span class="n">dir</span><span class="p">)</span>
<span class="n">setCurrentDir</span> <span class="s">"builds"</span>
<span class="k">if</span> <span class="n">os</span> <span class="o">==</span> <span class="s">"windows"</span><span class="p">:</span>
<span class="k">if</span> <span class="n">execShellCmd</span><span class="p">(</span><span class="s">interp"zip -9r ${dirName}.zip ${dirName}"</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="n">quit</span> <span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="n">execShellCmd</span><span class="p">(</span><span class="s">interp"tar cfz ${dirName}.tar.gz ${dirName}"</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span> <span class="n">quit</span> <span class="mi">1</span>
<span class="n">setCurrentDir</span> <span class="s">".."</span></code></pre></figure>
<p>Linux users have to install sdl2, sdl2_image, sdl2_ttf using their package
manager. Windows users get them bundled. Our build script creates this
directory structure when we run it with <code class="language-plaintext highlighter-rouge">nim -r c release</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.
├── platformer_1.0_linux_x86
│ ├── data
│ │ ├── default.map
│ │ ├── DejaVuSans.ttf
│ │ ├── grass.png
│ │ └── player.png
│ └── platformer
├── platformer_1.0_linux_x86.tar.gz
├── platformer_1.0_linux_x86_64
│ ├── data
│ │ ├── default.map
│ │ ├── DejaVuSans.ttf
│ │ ├── grass.png
│ │ └── player.png
│ └── platformer
├── platformer_1.0_linux_x86_64.tar.gz
├── platformer_1.0_win32
│ ├── data
│ │ ├── default.map
│ │ ├── DejaVuSans.ttf
│ │ ├── grass.png
│ │ └── player.png
│ ├── libfreetype-6.dll
│ ├── libpng16-16.dll
│ ├── platformer.exe
│ ├── SDL2.dll
│ ├── SDL2_image.dll
│ ├── SDL2_ttf.dll
│ └── zlib1.dll
├── platformer_1.0_win32.zip
├── platformer_1.0_win64
│ ├── data
│ │ ├── default.map
│ │ ├── DejaVuSans.ttf
│ │ ├── grass.png
│ │ └── player.png
│ ├── libfreetype-6.dll
│ ├── libpng16-16.dll
│ ├── platformer.exe
│ ├── SDL2.dll
│ ├── SDL2_image.dll
│ ├── SDL2_ttf.dll
│ └── zlib1.dll
└── platformer_1.0_win64.zip
</code></pre></div></div>
<p>Resulting downloads here: <a href="/public/platformer/platformer_1.0_win64.zip">Win64</a>, <a href="/public/platformer/platformer_1.0_win32.zip">Win32</a>, <a href="/public/platformer/platformer_1.0_linux_x86_64.tar.gz">Linux x86_64</a>, <a href="/public/platformer/platformer_1.0_linux_x86.tar.gz">Linux x86</a></p>
<h2 id="final-words">Final Words</h2>
<p>Somehow my articles are getting longer and this journey towards writing a
simple platform game took a few more explanations than expected. I hope this
article wasn’t too long and was instructive and helpful to get an understanding
of how to write platform games in Nim with SDL2.</p>
<p>All the material for this article is available in the <a href="https://github.com/def-/nim-platformer">repository on
GitHub</a>.
Because I changed around things late into the article I might have made a
mistake or missed something. If you find a bug or have a comment you can drop
me an email at <a href="mailto:dennis@felsing.org">dennis@felsing.org</a>.</p>
<p>Discussions on <a href="https://www.reddit.com/r/programming/comments/4o0h0u/writing_a_2d_platform_game_in_nim_with_sdl2/">r/programming</a> and <a href="https://news.ycombinator.com/item?id=11927780">Hacker News</a>.</p>
Experiences of Running an Online Game for 3 Years2016-06-09T00:00:00+02:00https://hookrace.net/blog/ddnet-evolution-architecture-technology
<p>It’s been roughly 3 years since <a href="https://ddnet.org/">DDraceNetwork</a> (DDNet)
started in the summer of 2013. Last year I wrote a non-technical <a href="https://forum.ddnet.org/viewtopic.php?f=3&t=1824">History of
DDNet</a>. Today in this post we
will dive into the technical side of what makes DDNet run.</p>
<p>For the uninitiated, DDNet is an <a href="https://github.com/ddnet/ddnet">open-source
modification</a> of
<a href="https://www.teeworlds.com/">Teeworlds</a>, a retro multiplayer shooter. DDNet is
a game in which you, instead of killing each other, cooperate with each other
to work your way through challenging maps, trying to beat the map or get a
better time than other teams.</p>
<!--more-->
<p>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/xwyk4hPZM1g" frameborder="0" allowfullscreen=""></iframe>
</div>
</p>
<p>What we offer for DDNet is the <a href="https://ddnet.org/downloads/">client and server
software</a> as well as official servers that run all
around the world. We have an international community of players trying to beat
the latest maps and records, and dozens of mappers who send in their newest
creations for our consideration. A new map is <a href="https://ddnet.org/releases/">released every few
days</a> and occasionally
<a href="https://ddnet.org/tournaments/">tournaments</a> happen in which the best compete
against each other on brand-new maps.</p>
<p>At the time of writing there are 968 people playing Teeworlds, 678 of which are
on a server running the DDNet mod, and 430 on the official DDNet servers. So
this project is not especially big and not many people stumble upon DDNet. Our
player base consists in a large part of old Teeworlds players and their friends
who heard about us by word of mouth.</p>
<p>Nevertheless I hope that the technical challenges we faced and face as well as
our solutions are interesting. Let’s start with an overview of what we have:</p>
<h2 id="overview--financing">Overview & Financing</h2>
<p>DDNet runs official servers in Germany, Russia, USA, Canada, Chile, Brazil,
China, South Africa and Iran.</p>
<p><img src="/public/locations.png" alt="DDNet Locations" /></p>
<p>Since DDNet is totally free, has no advertisements and no in-game purchases, it
offers no stream of revenue and is solely <a href="https://ddnet.org/funding/">funded by donations, donated servers
and my own money</a>. So one of the goals is to keep
costs down: On average we pay 10 € per month for each location.</p>
<p>Let’s see what we can offer with this to our thousands of players (<a href="https://ddnet.org/stats/">detailed
statistics</a>) and how DDNet evolved to keep with
increasing number of players, maps and the product of both, ranks.</p>
<p><a href="https://ddnet.org/funding/"><img src="/public/stats-finishes.png" alt="Number of finishes" /></a>
<a href="https://ddnet.org/funding/"><img src="/public/stats-players.png" alt="Number of players" /></a></p>
<h2 id="servers--locations">Servers & Locations</h2>
<p>We exclusively use cheap virtual private servers (VPS). They offer enough
performance for us and are significantly cheaper than dedicated servers. To
have a bit stronger guarantees on performance I prefer KVM and XEN over OpenVZ,
but there are good and bad hosters for each. In the end you always have to try
out a hoster for a few days up to a month to find out if it is suitable for
hosting official DDNet servers. The most important criteria for us are:</p>
<ul>
<li>Good and consistent CPU performance: should do about 400 players per CPU core</li>
<li>Low network latency to our players in the region: DDNet is only enjoyable
with a low and stable ping</li>
<li>0.5 to 1 GB of RAM is enough for <del>everyone</del> most locations: running
about 20 game servers per location</li>
</ul>
<p>When searching for hosters in a country you should prefer to look in the
language of the country instead of English. These tend to be cheaper and less
overcrowded by international buyers. Be prepared to use a translator and have
trouble with the support, but after some time all these hoster websites start
to look the same to you, no matter if they’re in English, German, French,
Spanish or Farsi.</p>
<p>When in doubt, ask locals what hosters they use or recommend, or check the
whois entries for servers in the region that seem to run well. For example the
<code class="language-plaintext highlighter-rouge">whois 31.186.251.128</code> entry tells you that <a href="https://ddnet.org/">DDNet.tw</a> (and
this blog) are hosted at Nuclear Fallout Enterprise, connected by the InterNAP
Network:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to '31.186.250.0 - 31.186.251.255'
% Abuse contact for '31.186.250.0 - 31.186.251.255' is 'noc@internap.com'
inetnum: 31.186.250.0 - 31.186.251.255
netname: INAP-LON-nuclearfallout-31-186-250-0
descr: Nuclear Fallout Enterprise
country: DE
admin-c: INAP-RIPE
tech-c: INAP-RIPE
status: ASSIGNED PA
mnt-by: INAP-MAINT-RIPE
created: 2013-08-08T19:09:47Z
last-modified: 2014-03-06T23:07:27Z
source: RIPE # Filtered
person: InterNAP Network Operator
address: InterNAP Network Services
address: Two Union Square
address: 601 Union Street Suite 1000
address: Seattle, WA 98101 USA
phone: +1 206 441 8800
fax-no: +1 206 256 9580
nic-hdl: INAP-RIPE
mnt-by: INAP-MAINT-RIPE
created: 2002-03-04T12:57:06Z
last-modified: 2002-03-04T12:57:06Z
source: RIPE # Filtered
% This query was served by the RIPE Database Query Service version 1.87.3 (DB-2)
</code></pre></div></div>
<p>Unfortunately the story doesn’t end here for us. Since DDNet is a multiplayer
open-source game we seem to attract the kind of people who know how to find a
DDoS (distributed denial of service) booter. This resulted in us being kicked
out at many hosters since they either do not have any (D)DoS protection or it
is not sufficient against some attacks, which then also affect other customers
of theirs.</p>
<p>In the end we settled on <a href="https://www.nfoservers.com/">NFOservers</a> for our main
servers, While they do not advertise any DDoS protection, they turned out to be
the only one of dozens of hosters I tried that can withstand most of the
attacks we get regularly. I’m of course talking about cheap VPSes, I hear that
good DDoS protection exists for a few hundred euros per month, which is far
outside of our reach. Unfortunately NFOservers is only available in USA and
Germany.</p>
<p>Here’s what the attacks in a regular 2 week period looks like in our most
popular location, Europe (GER server):</p>
<p>
<div style="height: 20em; overflow: scroll;">
<a href="/public/ddos.png"><img src="/public/ddos.png" alt="GER DDoS attacks" /></a>
</div>
</p>
<p>In Russia (Moscow) we had ok experiences with <a href="https://www.reg.com/">reg.com</a>,
at least the ping for local players is lower than in any other location.</p>
<p>Recently we got a server hosted in Iran again, which has always been a
difficult country to host a server in, for matters of cost as well as their
preference of an intranet over the internet.</p>
<p>In Chile we’re now with <a href="http://www.zglobalhost.com/">zGlobalHost</a>, one of the
few hosters with unlimited bandwidth and by far the best I found for just 9 €
per month. Their server room looks very cozy:</p>
<p><img src="/public/zglobalhost.jpg" alt="ZGlobalHost data center" /></p>
<p>In Chile we also had this <a href="https://www.facebook.com/zglobalhost/">bizarre
story</a> back in January, after the entire
data center of ZGlobalHost went down:</p>
<ul>
<li>
<p>January 26, 16:04 (local time Chile)<br />
Our provider GTD has trouble again and is still not giving estimated time of
repair. We will inform you with more news as soon as possible.</p>
</li>
<li>
<p>January 26, 18:06<br />
Our provider informs us that in the next 30 minutes everything should be
operational again.</p>
</li>
<li>
<p>January 26, 19:09<br />
GTD reports that there are two fiber cuts, one in Pudahuel and another in the
Eastern Zone. They are certifying the fibers and lifting services one by one.</p>
</li>
<li>
<p>January 26, 22:34<br />
Be informed that when we are back online and GTD reports that it will provide
further details about the incident as it was just reported that the fiber cut
was due to vandalism, the streets in Providencia are now reported to be safe
for repair works again.<br />
We do not have backup lines. It will at least take another hour but
repairs should not take longer than until this morning. You will be
mailed a report instantly when the servers are back online and tomorrow
GTD will send a report with details.</p>
</li>
<li>
<p>January 26, 22:36<br />
The new details we have say that there were 4 cuts today, which is why
there was such a long delay:<br />
Locations: Pudahuel, Providencia, La Florida, Macul<br />
Causes for each cut:
Fire, Vandalism, Highway Accident, Material fatigue in electric pole cut it down<br />
All ISPs in various areas have been affected, particularly fiber optic, both
business and home addresses.</p>
</li>
<li>
<p>January 27, 03:00<br />
Service is fully operational again since 02:30 AM.</p>
</li>
</ul>
<p>To make sure that our servers keep running well, we <a href="https://ddnet.org/status/">monitor
them</a> and record the <a href="https://ddnet.org/stats/server/">server
statistics</a>. When an unexpected event happens,
like a server downtime or high network traffic, a notification is automatically
sent. You can read more about this system in <a href="https://hookrace.net/blog/server-statistics/">a separate post from last
month</a>.</p>
<h2 id="software-platform">Software Platform</h2>
<p>As the operating system our servers run Linux, preferably
<a href="https://www.debian.org/">Debian</a>, but also some instances of Ubuntu and
CentOS. In areas like South Africa, South America and Iran bandwidth can be
exorbitantly expensive, so when you get a donated server in one of those
locations you can not afford to be picky.</p>
<p>For the databases we use <a href="https://mariadb.org/">MariaDB</a>, the drop-in MySQL
fork started by the original MySQL developers after Oracle acquired it. We use
this database to record our maps, saved games and the ranks of all players
worldwide and keep these synchronized around the world.</p>
<p>Initially we started with a single VPS and a database running directly on it.
As we grew to add more locations around the world we used a circular
master-slave replication. With time some servers turned out to be less stable
than others, so they were taken out of the loop, their game servers
communicating with nearby MariaDB servers instead:</p>
<p><img class="halfimg" src="/public/ddnet-replication-1.png" alt="Old MariaDB Replication" /><img class="halfimg" src="/public/ddnet-replication-2.png" alt="New MariaDB Replication" /></p>
<p>Now we’re switching away from this circular master-slave replication to having
a small number of master databases which are replicated by slaves at each
location. This has the advantage that database reads happen locally, so getting
your own rank is always fast. Meanwhile new ranks are sent to a master server
when possible or otherwise stored in a local file to be added later. This is
necessary since our cheap servers are not always connected to the entire
internet, sometimes they are only available from inside their own country.</p>
<p>The DDNet client and server, being based on Teeworlds, are also written in
C/C++ (also known as C with classes). Luckily their performance was good from
the start, but a few optimizations still helped with keeping the load low when
running up to 60 game servers on a single VPS, for example:</p>
<ul>
<li>Reduce the number of syscalls by caching the value of <code class="language-plaintext highlighter-rouge">gettimeofday()</code> until
a new tick happens or network packet comes in</li>
<li>Do not execute the main game loop when the server is currently empty, as
nothing will happen with no players anyway</li>
<li>Use CPU (<code class="language-plaintext highlighter-rouge">nice</code>) and IO (<code class="language-plaintext highlighter-rouge">ionice</code>) priorities to prevent background tasks
(lower priority) to get in the way of game server processes (high priority)</li>
<li>Each VPS compiles its own server binary tuned for the specific target
architecture</li>
<li>Prefer to download map files over HTTP from our web server instead of the
built-in UDP protocol that can be shaky and slow. This also reduces the
network traffic on game servers, keeping network jitter lower.</li>
<li>Instead of downloading the entire archive on a new client release, update
only the changed files</li>
</ul>
<p>Generally the idea is to wait until you notice that some resource is being used
too much, benchmark and analyze what the reason for it is, and only then
optimize this.</p>
<h2 id="website">Website</h2>
<p>Back when DDNet started in 2013 the website ran on the same VPS as our game
servers. Later we switched to hosting the website on a separate VPS, in part to
improve the DDoS protection by having mostly UDP on game servers and mostly TCP
on the web server, but also to make sure that the website traffic and CPU usage
do not impact the game servers.</p>
<p>This has become even more important since the map downloads are now also
happening over HTTP from the web server instead of going over UDP from the game
servers. Automatically updating the list of available maps is done with a small
Nim script that I already presented in <a href="/blog/what-makes-nim-practical/#use-as-a-scripting-language">a previous
article</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="c">#!/usr/bin/env nimscript</span>
<span class="k">import</span> <span class="n">os</span><span class="p">,</span> <span class="n">crc32</span><span class="p">,</span> <span class="n">strutils</span>
<span class="k">const</span>
<span class="n">baseDir</span> <span class="o">=</span> <span class="s">"/home/teeworlds/servers"</span>
<span class="n">mapdlDir</span> <span class="o">=</span> <span class="s">"/var/www-maps"</span>
<span class="k">for</span> <span class="n">kind</span><span class="p">,</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">walkDir</span> <span class="n">baseDir</span><span class="o">/</span><span class="s">"maps"</span><span class="p">:</span>
<span class="k">if</span> <span class="n">kind</span> <span class="o">!=</span> <span class="n">pcFile</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">let</span> <span class="p">(</span><span class="n">dir</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">ext</span><span class="p">)</span> <span class="o">=</span> <span class="n">splitFile</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ext</span> <span class="o">!=</span> <span class="s">".map"</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">let</span>
<span class="n">sum</span> <span class="o">=</span> <span class="n">crc32FromFile</span><span class="p">(</span><span class="n">path</span><span class="p">).</span><span class="kt">int64</span><span class="p">.</span><span class="n">toHex</span><span class="p">(</span><span class="mi">8</span><span class="p">).</span><span class="n">toLower</span>
<span class="n">newName</span> <span class="o">=</span> <span class="n">name</span> <span class="o">&</span> <span class="s">"_"</span> <span class="o">&</span> <span class="n">sum</span> <span class="o">&</span> <span class="n">ext</span>
<span class="n">newPath</span> <span class="o">=</span> <span class="n">mapdlDir</span> <span class="o">/</span> <span class="n">newName</span>
<span class="n">tmpPath</span> <span class="o">=</span> <span class="n">newPath</span> <span class="o">&</span> <span class="s">".tmp"</span>
<span class="k">if</span> <span class="n">existsFile</span> <span class="n">newPath</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">copyFile</span> <span class="n">path</span><span class="p">,</span> <span class="n">tmpPath</span>
<span class="n">moveFile</span> <span class="n">tmpPath</span><span class="p">,</span> <span class="n">newPath</span></code></pre></figure>
<p>Most of the <a href="https://ddnet.org/">DDNet.tw</a> website is <a href="https://github.com/ddnet/ddnet-web">statically built by
jekyll</a> and automatically deploying using
GitHub’s webhooks. That’s a simple solution and requires very few resources.</p>
<p>Pages about <a href="https://ddnet.org/status/">player status</a>,
<a href="https://ddnet.org/ranks/">ranks</a>, <a href="https://ddnet.org/releases/">map releases</a>
and <a href="https://ddnet.org/mappers/">mappers</a> are also statically built, but by
<a href="https://github.com/ddnet/ddnet-scripts/tree/master/servers/scripts">Python
scripts</a>
directly on the server. This also makes sense since these pages are expensive
to build and are requested more often than they have to be generated.</p>
<p>We also used to generate the <a href="https://ddnet.org/players/">player pages</a> in the
same way, but since we have about 90,000 ranked players so far this would’ve
become a bit too expensive. Instead we run a small uWSGI Python server to
dynamically generate them now from the data kept in memory. Right now it takes
about 40-80 ms to generate a single player page, by far fast enough for us. If
the computation time goes up or we suddenly become much more popular caching
could be implemented.</p>
<p>We used to package a growing selection of skins with the DDNet client, but
maintaining this became too cumbersome and increased memory usage for the
client. Now instead the website offers a <a href="https://ddnet.org/skins/">database of
skins</a> where you can select individual skins or skin
packs.</p>
<h2 id="software-releases">Software Releases</h2>
<p>DDNet has been in active development for the last 3 years, with our first
client & server software release back in <a href="https://github.com/ddnet/ddnet/releases/tag/1.18">October 9,
2013</a>.</p>
<p>Initially I used to build the releases manually using virtual machines for
Windows and Linux, later also Mac OS X. After some time this got tedious of
course, so I wrote an <a href="https://github.com/ddnet/ddnet-scripts/blob/master/ddnet-release.sh">automated build
script</a>.
It uses a Debian chroot for Linux building, MinGW for Windows and a QEMU VM for
Mac OS X. I set up the Mac OS X QEMU image using <a href="http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/">a continuously updated
guide</a> and it works just
fine. I connect to the guest OS through a simple ssh connection that is
forwarded to the virtual machine:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Start the Mac OS X VM</span>
qemu-system-x86_64 <span class="o">[</span>...] <span class="se">\</span>
<span class="nt">-netdev</span> user,id<span class="o">=</span>hub0port0,hostfwd<span class="o">=</span>tcp::10022-:22 <span class="se">\</span>
<span class="nt">-device</span> e1000-82545em,netdev<span class="o">=</span>hub0port0,id<span class="o">=</span>mac_vnet0 <span class="se">\</span>
&>/dev/null &
<span class="c"># Wait until VM is booted up</span>
<span class="k">while</span> <span class="o">!</span> ssh <span class="nt">-p</span> 10022 <span class="nt">-o</span> <span class="nv">ConnectTimeout</span><span class="o">=</span>10 localhost <span class="nb">exit</span><span class="p">;</span> <span class="k">do </span><span class="nb">true</span><span class="p">;</span> <span class="k">done</span>
<span class="c"># Run build script</span>
ssh <span class="nt">-p</span> 10022 localhost <span class="s2">"# Run build script for Mac OS X"</span></code></pre></figure>
<p>Building for Android is also possible but disabled right now because we
still have to port the Android version to SDL2, and DDNet is not really
playable on Android anyway since a mouse is the most important tool in the
game.</p>
<p>When you think about it, it’s pretty amazing that you can easily cross-compile
on a single machine for all of these targets:</p>
<ul>
<li>Windows x86, x86_64</li>
<li>Linux x86, x86_64</li>
<li>Mac OS X</li>
<li>Android armeabi, armeabi-v7a, mips, x86</li>
</ul>
<p>The automated builds happen on my cheap ASRock Q1900-ITX home sever (which was
also used to <a href="/blog/ddnet-live/">stream gameplay from DDNet servers</a>) and
still only take a few minutes to complete:</p>
<p><img src="/public/q1900-itx.jpg" alt="Home server" /></p>
<p>Finally when all goes well you get a small summary of the build times that
looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Preparation: 10 s
Linux x86_64: 62 s
Linux x86: 64 s
Windows x86_64: 106 s
Windows x86: 84 s
</code></pre></div></div>
<p>For the automated builds we use bundled static libraries, while people who
build our software locally probably prefer to use their system libraries. Using
<code class="language-plaintext highlighter-rouge">pkg-config</code> we detect whether libraries are available at compile-time and
otherwise use the static ones if they are available. This behaviour can be
controlled manually as well:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Analog to ./configure</span>
bam config curl.use_pkgconfig<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
opus.use_pkgconfig<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
opusfile.use_pkgconfig<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
ogg.use_pkgconfig<span class="o">=</span><span class="nb">false</span>
<span class="c"># Analog to make</span>
bam release</code></pre></figure>
<p>Even a <a href="http://teewebs.net/">JavaScript port</a> of DDNet client is available for
playing without any installation, compiled with Emscripten. For this purpose
the official servers also accept WebSocket connections additionally to UDP.</p>
<p>New releases are added to the <a href="https://ddnet.org/downloads/">website</a> with a
nice changelog. But most existing players get their update notification in the
client directly and run the updater from there, only downloading the files that
actually changed instead of the entire new release.</p>
<h2 id="map-testing--releases">Map Testing & Releases</h2>
<p>The best part about DDNet is its active community. New maps for our servers are
regularly sent in to be tested and later released. Initially the testing
process was implemented as <a href="https://trac.edgewall.org/">Trac</a> installation. But
it turned out that the non-technical people did not appreciate it much, so we
switched to a <a href="https://forum.ddnet.org/">regular phpBB forum</a> for testing and
communication.</p>
<p>New maps can be uploaded by our testers to the test servers and then tested
there. The map upload happens with an upload script to the web server. The test
servers around the world use sshfs to access these maps.</p>
<p>Once a new map has been cleared by the testers and all bugs fixed by the
mapper, it can be
<a href="https://github.com/ddnet/ddnet-scripts/blob/master/map-release">released</a> on
the proper server. The preparation for this happens on the web server and is
then synchronized to the other servers using an internal git repository.</p>
<p>The new map is also announced on the <a href="https://ddnet.org/releases/">recent map releases
page</a> as well as its feed. Players on the official
servers are informed about the map release using a broadcast message.</p>
<p>It’s possible to take a look at our maps using a WebGL renderer that supports
parts of the map format, but no animations for example: <a href="https://ddnet.org/maps/?map=Lonely">Lonely
map</a></p>
<p>When a popular new map is released, it’s possible to have hundreds of players
taking up the challenge of playing the new map at once, either in small teams
or in big groups.</p>
<p>Finally every day the newly released maps are added to our <a href="https://github.com/ddnet/ddnet-maps">public maps
repository</a>,
which can be used to easily run a clone of the official DDNet servers at home or
on your own server. Just download the repository, add the DDNet-Server binary
to the same directory and run it. A simple shell script suffices to update this
repository:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/usr/bin/env zsh</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /home/teeworlds/ddnet-maps
<span class="nb">cd</span> /home/teeworlds/ddnet-maps
<span class="nb">rm</span> <span class="nt">-rf</span> types
<span class="nb">echo</span> <span class="s1">'add_path $USERDIR\nadd_path $CURRENTDIR'</span> <span class="o">></span> storage.cfg
<span class="k">for </span>i <span class="k">in</span> <span class="sb">`</span><span class="nb">cat</span> ../servers/all-types<span class="sb">`</span><span class="p">;</span> <span class="k">do
</span><span class="nv">TYPEDIR</span><span class="o">=</span>types/<span class="k">${</span><span class="nv">i</span>:l<span class="k">}</span>
<span class="nb">echo</span> <span class="s2">"add_path </span><span class="nv">$TYPEDIR</span><span class="s2">"</span> <span class="o">>></span> storage.cfg
<span class="nb">mkdir</span> <span class="nt">-p</span> <span class="nv">$TYPEDIR</span>/maps
<span class="nb">grep</span> <span class="s2">"|"</span> ../servers/<span class="nv">$TYPEDIR</span>/maps | <span class="nb">cut</span> <span class="nt">-d</span><span class="s2">"|"</span> <span class="nt">-f2</span> | <span class="k">while </span><span class="nb">read </span>j<span class="p">;</span> <span class="k">do
</span><span class="nb">cp</span> <span class="nt">--</span> <span class="s2">"../servers/maps/</span><span class="nv">$j</span><span class="s2">.map"</span> <span class="nv">$TYPEDIR</span>/maps
<span class="nb">cp</span> ../servers/<span class="nv">$TYPEDIR</span>/flexvotes.cfg <span class="nv">$TYPEDIR</span>
<span class="nb">grep</span> <span class="nt">-v</span> <span class="s2">"flexname.cfg"</span> ../servers/<span class="nv">$TYPEDIR</span>/flexreset.cfg <span class="o">></span> <span class="nv">$TYPEDIR</span>/flexreset.cfg
<span class="nb">tail</span> <span class="nt">-n</span> +5 ../servers/<span class="nv">$TYPEDIR</span>/votes.cfg <span class="o">></span> <span class="nv">$TYPEDIR</span>/votes.cfg
<span class="k">done
done
</span>git add <span class="k">*</span> &>/dev/null
git commit <span class="nt">-a</span> <span class="nt">-m</span> <span class="s2">"daily update"</span> &>/dev/null
git push &>/dev/null</code></pre></figure>
<h2 id="server-types">Server Types</h2>
<p>In the start DDNet hosted only new maps categorized by their difficulty:</p>
<ul>
<li><a href="https://ddnet.org/ranks/novice/">Novice</a></li>
<li><a href="https://ddnet.org/ranks/moderate/">Moderate</a></li>
<li><a href="https://ddnet.org/ranks/brutal/">Brutal</a></li>
</ul>
<p>Later we also added categories for maps from old servers:</p>
<ul>
<li><a href="https://ddnet.org/ranks/oldschool/">Oldschool</a>: Ancient maps</li>
<li><a href="https://ddnet.org/ranks/ddmax/">DDmaX</a>: Old DDRace server before DDNet</li>
</ul>
<p>As well as categories for different kinds of single player maps:</p>
<ul>
<li><a href="https://ddnet.org/ranks/solo/">Solo</a></li>
<li><a href="https://ddnet.org/ranks/race/">Race</a>: Classic mod with fewer features,
converted maps to new DDNet format</li>
<li><a href="https://ddnet.org/ranks/dummy/">Dummy</a>: Maps where you play with another
player that is unable to move</li>
</ul>
<p>The idea is that, since we are the major hoster in many countries, many of
these good maps would be forgotten forever if we don’t host them. At the same
time we try to motivate people to make creative new maps, having worked
together with mappers to add new features that they require.</p>
<p>While the DDNet servers are the main focus, we also use our servers for hosting
a few other Teeworlds mods since they use very few resources and the servers
are running already anyway.</p>
<h2 id="tournaments">Tournaments</h2>
<p>When a well-known mapper makes a special map, they can choose to have their map
played at an official DDNet <a href="https://ddnet.org/tournaments/">tournament</a>. For
this it is necessary to keep the map under wraps, showing it only to a few
select testers. This ensures that none of the players at the tournament will be
familiar with the map already, which would give them an unfair advantage.</p>
<p>Usually tournaments are done on Sunday, 20:00 Central European Time since most
of our player base is located in Europe. You can read about the work that goes
into <a href="https://github.com/ddnet/ddnet-scripts/blob/master/tournament-preparation">preparing a
tournament</a>.
But the really stressful part is actually holding the tournament.</p>
<p>Since it runs at the same time on all (currently 9-10) servers worldwide, you
need to control them all at once. If the tournament map has not been tested
well enough it might need to be quickly fixed and reloaded during the
tournament. Sometimes problems occur on a single server. Unfortunately
tournaments also attract DDoS attacks.</p>
<p>Here you can see the last tournament’s top run, for which the players had 2
hours to compete:</p>
<p>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/wzfSDzgP6mA" frameborder="0" allowfullscreen=""></iframe>
</div>
</p>
<p>Some of our mappers even created <a href="https://forum.ddnet.org/viewtopic.php?p=1560#p1560">fancy ingame loading
screens</a> for the start of
the tournament:</p>
<p>
<div class="video-container">
<iframe src="https://www.youtube.com/embed/_OEgiJFKtQw" frameborder="0" allowfullscreen=""></iframe>
</div>
</p>
<h2 id="client--server-development">Client & Server Development</h2>
<p>Development of the client and server software is often challenging because of
backwards compatibility:</p>
<ol>
<li>People with regular Teeworlds client should still be able to play on DDNet</li>
<li>People with old DDNet client (or custom client) should still be able to play</li>
<li>Old ranks would be invalidated by slight fixes/changes in game physics</li>
</ol>
<p>Nevertheless we managed to put in many new features, for example:</p>
<ul>
<li>64 player servers instead of just 16 players:<br />
Players with vanilla Teeworlds client only see the 15 closest players on a server.</li>
<li>Protect players from connection timeouts:<br />
The timeout protection works by the client having a unique timeout code that
it sends to the server whenever it connects. This tells the server if the
player has had a timeout before and they can get back their ingame character.</li>
<li>Allow saving game in team and continuing at a later time:<br />
The save games are stored in our MariaDB database, but can only be loaded at
a single location. Otherwise players would have the ability to load a save
game multiple times, which is not what we want.</li>
<li>Save the ranks of teams and not just single players</li>
<li>SDL2 client instead of old SDL1.2</li>
<li>New physics: Jetpack, extended teleporters, walljump, tune zones</li>
<li>Highly compressed Opus sounds in maps, it’s amazing how much quality you get
with small bit rates</li>
<li>In-client updater using HTTPS and a <a href="http://update2.ddnet.org/update.json">simple JSON
file</a> for control</li>
<li>Dummy feature to control two ingame characters at once:<br />
At first this was meant to help map testers to be able to test maps easier,
without requiring a second player. But it’s also very popular with regular
players as well as mappers who come up with interesting maps where only one
player has all the control.</li>
<li>…</li>
</ul>
<p>Particularly important for the automation of the official DDNet servers is our
FIFO console system. Every server listens for commands from a FIFO file. This
enables us to automate the map releases, server restarts and broadcasts among
other things. To broadcast a message about a new map release to all servers,
you can just do this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">echo</span> <span class="s1">'broadcast "New map Uniswim by Im corneum just released on Solo server"'</span> <span class="o">></span> servers/<span class="k">*</span>.fifo</code></pre></figure>
<p>We have a <a href="https://github.com/ddnet/ddnet-scripts/blob/master/servers/bc.py">small Python
script</a> that
can do this automatically and also broadcast in a <a href="https://github.com/ddnet/ddnet-scripts/blob/master/servers/scripts/asciiart">custom big ASCII art
font</a>:</p>
<p><img src="/public/ascii-ddnet.png" alt="ASCII DDNET" /></p>
<p>Using the FIFO system at multiple locations at once is achieved using Cluster SSH.</p>
<p>We also offer a list of official DDNet servers <a href="http://update2.ddnet.org/ddnet-servers.json">in
JSON</a> which is used to inform the
client of the official servers as well as internally to generate the <a href="https://ddnet.org/status/">status
page</a>.</p>
<p>Unfortunately attacks with spoofed IP addresses are quite common for us. So we
have added a spoofing protection using tokens to prevent attackers from filling
up servers with players with spoofed IPs.</p>
<p>We have no automated tests for the client and server, the community quickly
catches new bugs though. At least we use <a href="https://circleci.com/gh/ddnet/ddnet">automated Circle
CI</a> building to make sure new changes
still compile fine.</p>
<h2 id="communication">Communication</h2>
<p>Ingame communication is a bit tricky since the game is built on simplicity and
we prefer not to have accounts. Introducing accounts this late into existance
would give another vector of abuse by giving attackers the chance to register
other people’s names. So you can never be sure who you’re actually talking to.
Thanks to some Social Engineering this has been a common source of leaked
moderator passwords.</p>
<p>So for official communications the <a href="https://forum.ddnet.org/">DDNet forum</a> is
used instead. We tried to use <a href="https://letsencrypt.org/">Let’s Encrypt</a> for
SSL/TLS certificates for the website, but it turned out that Windows XP is <a href="https://github.com/certbot/certbot/issues/1660">not
supported</a> and many of our
players have old systems still running Windows XP. I guess that’s in the nature
of running one of the few actively developed online games that work perfectly
fine on 15 year old machines. Instead we’re now using
<a href="https://www.startssl.com/">StartSSL</a>, which seems to work fine, except that
wildcards would cost money. Once Let’s Encrypt works fine on Windows XP we
might be able to switch back.</p>
<p>Otherwise technically inclined people mostly discuss on IRC
(<a href="irc://irc.quakenet.org/ddnet">#ddnet</a> on QuakeNet,
<a href="http://webchat.quakenet.org/?channels=ddnet&uio=d4">WebChat</a>,
<a href="https://ddnet.org/irclogs/">Logs</a>) and Skype. Most non-technical discussions
happen via Skype.</p>
<h2 id="future-directions">Future Directions</h2>
<p>The main goal is to keep DDNet running, well maintained, and keep a steady
stream of new maps from the community.</p>
<p>We submitted DDNet to <a href="http://steamcommunity.com/sharedfiles/filedetails/?id=506147661">Steam
Greenlight</a>
and have been greenlit since then, which means we could release it on Steam
now. This would be an opportunity for us to reach more potential players,
especially those who have never played Teeworlds before. The main problem right
now is that DDNet is a difficult game to get started with, since it requires
practice, fine control and patience. Right now work is <a href="https://forum.ddnet.org/viewtopic.php?t=3771">being
done</a> to create a tutorial for new
players and generally improve the ease of getting started.</p>
<h2 id="discussion">Discussion</h2>
<p>You can find discussions about this article on <a href="https://news.ycombinator.com/item?id=11874830">Hacker News</a>, <a href="https://www.reddit.com/r/programming/comments/4nnlsq/experiences_of_running_an_online_game_for_3_years/">r/programming</a> and also the <a href="https://forum.ddnet.org/viewtopic.php?f=3&t=3779">DDNet Forum</a>.</p>
Introduction to Metaprogramming in Nim2016-06-06T00:00:00+02:00https://hookrace.net/blog/introduction-to-metaprogramming-in-nim
<h2 id="introduction-to-the-introduction-meta-introduction">Introduction to the Introduction (Meta-Introduction)</h2>
<p><a href="https://en.wikipedia.org/wiki/Metaprogramming">Wikipedia</a> gives us a nice
description of metaprogramming:</p>
<blockquote>
<p>Metaprogramming is the writing of computer programs with the ability to
treat programs as their data. It means that a program could be designed to
read, generate, analyse and/or transform other programs, and even modify
itself while running.</p>
</blockquote>
<p>In this article we will explore <a href="http://nim-lang.org/">Nim</a>’s metaprogramming
capabilities, which are quite powerful and yet still easy to use. After all
great metaprogramming is one of Nim’s main features. The general rule is to use
the least powerful construct that is still powerful enough to solve a problem,
in this order:</p>
<!--more-->
<ol>
<li><a href="#normal-procs">Normal procs</a> and <a href="#inline-iterators">inline iterators</a></li>
<li><a href="#generic-procs">Generic procs</a> and <a href="#closure-iterators">closure iterators</a></li>
<li><a href="#templates">Templates</a></li>
<li><a href="#macros">Macros</a></li>
</ol>
<p>So before looking at Nim’s two main metaprogramming constructs, templates and
macros, we’ll look at what we can do with procs and iterators as well.</p>
<h2 id="regular-programming-constructs">Regular Programming Constructs</h2>
<h3 id="normal-procs">Normal procs</h3>
<p>We’re in normal programming land here. Regular procedures are what you know as
<em>functions</em> elsewhere and they’re pretty easy to define and use:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">sayHi</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="n">echo</span> <span class="s">"Hello "</span><span class="p">,</span> <span class="n">name</span>
<span class="n">sayHi</span><span class="p">(</span><span class="s">"World"</span><span class="p">)</span>
<span class="n">sayHi</span> <span class="s">"World"</span>
<span class="s">"World"</span><span class="p">.</span><span class="n">sayHi</span></code></pre></figure>
<h3 id="generic-procs">Generic procs</h3>
<p>With generics we can define procs that work on multiple types. Actually a new
proc will be generated based on our generic definition for each instantiation:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">min</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">T</span><span class="p">):</span> <span class="n">T</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">x</span> <span class="o"><</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="k">else</span><span class="p">:</span> <span class="n">y</span>
<span class="n">echo</span> <span class="n">min</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="c"># more explicitly: min[int](2, 3)</span>
<span class="n">echo</span> <span class="n">min</span><span class="p">(</span><span class="s">"foo"</span><span class="p">,</span> <span class="s">"bar"</span><span class="p">)</span> <span class="c"># min[string]("foo", "bar")</span></code></pre></figure>
<h3 id="inline-iterators">Inline iterators</h3>
<p>Inline iterators are the default iterators in Nim. They get compiled into high
performance loops:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">iterator</span> <span class="n">reverseItems</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="kt">char</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">countdown</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="n">high</span><span class="p">,</span> <span class="n">x</span><span class="p">.</span><span class="n">low</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="s">"foo"</span><span class="p">.</span><span class="n">reverseItems</span><span class="p">:</span>
<span class="n">echo</span> <span class="n">c</span></code></pre></figure>
<p>So this code gets compiled into:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">let</span> <span class="n">x</span> <span class="o">=</span> <span class="s">"foo"</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">countdown</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="n">high</span><span class="p">,</span> <span class="n">x</span><span class="p">.</span><span class="n">low</span><span class="p">):</span>
<span class="k">let</span> <span class="n">c</span> <span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span>
<span class="n">echo</span> <span class="n">c</span></code></pre></figure>
<p>Of course we can make iterators generic too:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">iterator</span> <span class="n">reverseItems</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">T</span><span class="p">):</span> <span class="n">auto</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">countdown</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="n">high</span><span class="p">,</span> <span class="n">x</span><span class="p">.</span><span class="n">low</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span></code></pre></figure>
<h3 id="closure-iterators">Closure iterators</h3>
<p>Inline iterators simultaneously have the advantage and disadvantage of being
translated into loops. This means you can not pass them around. This limitation
can be lifted by using closure iterators instead:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">math</span>
<span class="k">proc </span><span class="nf">powers</span><span class="p">(</span><span class="n">m</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="n">auto</span> <span class="o">=</span>
<span class="c">#return iterator: int {.closure.} = # Make a closure explicitly</span>
<span class="k">return</span> <span class="k">iterator</span><span class="p">:</span> <span class="kt">int</span> <span class="o">=</span> <span class="c"># Compiler makes this a closure for us</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="mf">0</span><span class="p">..</span><span class="kt">int</span><span class="p">.</span><span class="n">high</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">n</span><span class="p">^</span><span class="n">m</span>
<span class="k">var</span>
<span class="n">squares</span> <span class="o">=</span> <span class="n">powers</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="n">cubes</span> <span class="o">=</span> <span class="n">powers</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">4</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Square: "</span><span class="p">,</span> <span class="n">squares</span><span class="p">()</span> <span class="c"># 0, 1, 4, 9</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">4</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Cube: "</span><span class="p">,</span> <span class="n">cubes</span><span class="p">()</span> <span class="c"># 0, 1, 8, 27</span>
<span class="n">echo</span> <span class="s">"Square: "</span><span class="p">,</span> <span class="n">squares</span><span class="p">()</span> <span class="c"># 16</span>
<span class="n">echo</span> <span class="s">"Cube: "</span><span class="p">,</span> <span class="n">cubes</span><span class="p">()</span> <span class="c"># 64</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">squares</span><span class="p">():</span> <span class="c"># Go through all the remaining squares</span>
<span class="n">echo</span> <span class="s">"Square: "</span><span class="p">,</span> <span class="n">x</span> <span class="c"># 25, 36, 49, 64, ...</span></code></pre></figure>
<p>As you can see closure iterators keep their state. You can call them again and
get the next value, or use them inside of a for-loop to get out many values.</p>
<h2 id="templates">Templates</h2>
<p>You can think of templates as Nim’s equivalent to the C preprocessor. But
templates are written in Nim itself and fit well into the rest of the language.</p>
<p>Templates simply insert their code at the invocation site, working at the level
of the abstract syntax tree. They can be used in just the same way as procs.</p>
<h3 id="logger">Logger</h3>
<p>A common example are loggers, which we looked at in <a href="/blog/writing-an-async-logger-in-nim/">another
article</a> already. Consider that you want
to have extensive debug logging in your program. A trivial implementation would
look like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">times</span><span class="p">,</span> <span class="n">os</span>
<span class="k">type</span> <span class="n">Level</span><span class="o">*</span> <span class="p">{.</span><span class="n">pure</span><span class="p">.}</span> <span class="o">=</span> <span class="k">enum</span>
<span class="n">debug</span><span class="p">,</span> <span class="n">info</span><span class="p">,</span> <span class="n">warn</span><span class="p">,</span> <span class="n">error</span><span class="p">,</span> <span class="n">fatal</span>
<span class="k">var</span> <span class="n">logLevel</span><span class="o">*</span> <span class="o">=</span> <span class="n">Level</span><span class="p">.</span><span class="n">debug</span>
<span class="k">proc </span><span class="nf">debug</span><span class="o">*</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="p">`</span><span class="o">$</span><span class="p">`</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">logLevel</span> <span class="o"><=</span> <span class="n">Level</span><span class="p">.</span><span class="n">debug</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"[</span><span class="si">$#</span><span class="s"> </span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">getDateStr</span><span class="p">(),</span> <span class="n">getClockStr</span><span class="p">(),</span>
<span class="n">join</span> <span class="n">args</span><span class="o">]</span>
<span class="k">proc </span><span class="nf">expensiveDebuggingInfo</span><span class="o">*</span><span class="p">:</span> <span class="kt">string</span> <span class="o">=</span>
<span class="n">sleep</span><span class="p">(</span><span class="n">milsecs</span> <span class="o">=</span> <span class="mi">1000</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="s">"Everything looking good!"</span>
<span class="n">debug</span> <span class="n">expensiveDebuggingInfo</span><span class="p">()</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2016-06-05 22:00:50]: Everything looking good!
</code></pre></div></div>
<p>We have to call <code class="language-plaintext highlighter-rouge">expensiveDebuggingInfo</code> to get the debugging info, which is
fine right now since our <code class="language-plaintext highlighter-rouge">logLevel</code> is set to <code class="language-plaintext highlighter-rouge">Level.debug</code>. But it stops being
fine when we instead set <code class="language-plaintext highlighter-rouge">logLevel</code> to anything higher than <code class="language-plaintext highlighter-rouge">debug</code>. Then it
still takes a full second to evaluate the <code class="language-plaintext highlighter-rouge">expensiveDebuggingInfo</code> parameter
for <code class="language-plaintext highlighter-rouge">debug</code>, but inside of <code class="language-plaintext highlighter-rouge">debug</code> nothing is done with that information. This
is of course a consequence of call-by-value argument evaluation, which Nim
uses, just as most other languages do. A notable exception would be lazy
evaluation in Haskell, where this kind of logger would work perfectly fine,
only calling <code class="language-plaintext highlighter-rouge">expensiveDebuggingInfo</code> when its value is actually needed.</p>
<p>But let’s stay in Nim-land and use a template instead of a proc to magically
fix this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">debug</span><span class="o">*</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="p">`</span><span class="o">$</span><span class="p">`</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">logLevel</span> <span class="o"><=</span> <span class="n">Level</span><span class="p">.</span><span class="n">debug</span><span class="p">:</span>
<span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span>
<span class="n">echo</span> <span class="s">"[</span><span class="si">$#</span><span class="s"> </span><span class="si">$#</span><span class="s">][</span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">getDateStr</span><span class="p">(),</span> <span class="n">getClockStr</span><span class="p">(),</span>
<span class="n">module</span><span class="p">,</span> <span class="n">join</span> <span class="n">args</span><span class="o">]</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2016-06-05 22:01:30][logger]: Everything looking good!
</code></pre></div></div>
<p>Note that we also conveniently use <code class="language-plaintext highlighter-rouge">instantiationInfo()</code> to find out at what
location in the program our template was instantiated, something we could not
do using a procedure.</p>
<p>We can still call the template in the exact same way as the proc. But now we
have the advantage that the template is inlined at compiletime, so
<code class="language-plaintext highlighter-rouge">expensiveDebuggingInfo</code> is only called if the runtime <code class="language-plaintext highlighter-rouge">logLevel</code> actually
requires it. Perfect.</p>
<h3 id="safe-locking">Safe locking</h3>
<p>Another problem that can be solved with a template is automatically acquiring and releasing a system <a href="http://nim-lang.org/docs/locks.html">lock</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">locks</span>
<span class="k">template</span> <span class="n">withLock</span><span class="p">(</span><span class="n">lock</span><span class="p">:</span> <span class="n">Lock</span><span class="p">,</span> <span class="n">body</span><span class="p">:</span> <span class="n">untyped</span><span class="p">)</span> <span class="o">=</span>
<span class="n">acquire</span> <span class="n">lock</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">body</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">release</span> <span class="n">lock</span></code></pre></figure>
<p>Compile with <code class="language-plaintext highlighter-rouge">--threads:on</code> for platform independent lock support.</p>
<p>This looks pretty simple, we just acquire the lock, execute the passed
statements and finally release the lock, even if exceptions have been thrown.
We can pass any set of statements as the <code class="language-plaintext highlighter-rouge">body</code>. The usage is as easy as using
a built-in if statement:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">lock</span><span class="p">:</span> <span class="n">Lock</span>
<span class="n">initLock</span> <span class="n">lock</span>
<span class="n">withLock</span> <span class="n">lock</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Do something that requires locking"</span>
<span class="n">echo</span> <span class="s">"This might throw an exception"</span></code></pre></figure>
<p>When our template accepts a value of type <code class="language-plaintext highlighter-rouge">stmt</code> we can use the colon to pass an entire indented block of code. When we have multiple parameters of type <code class="language-plaintext highlighter-rouge">stmt</code> the <a href="http://nim-lang.org/docs/manual.html#procedures-do-notation">do notation</a> can be used.</p>
<p>This gets transformed into:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">lock</span><span class="p">:</span> <span class="n">Lock</span>
<span class="n">initLock</span> <span class="n">lock</span>
<span class="n">acquire</span> <span class="n">lock</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Do something that requires locking"</span>
<span class="n">echo</span> <span class="s">"This might throw an exception"</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">release</span> <span class="n">lock</span></code></pre></figure>
<p>Now we will never forget to call <code class="language-plaintext highlighter-rouge">release lock</code>. You could use this to make a
higher level locking library that only exposes <code class="language-plaintext highlighter-rouge">withLock</code> instead of the
lower-level <code class="language-plaintext highlighter-rouge">acquire</code> and <code class="language-plaintext highlighter-rouge">release</code> primitives.</p>
<h2 id="macros">Macros</h2>
<p>Just like templates, macros are executed at compiletime. But with templates you
can only do constant substitutions in the AST. With macros you can analyze the
passed arguments and create a new AST at the current position in any way you
want. A nice property of Nim is that these compiletime macros are also written
in the regular Nim language, so there is no need to learn another language.</p>
<p>A simple way to create an AST is to use <code class="language-plaintext highlighter-rouge">parseStmt</code> and <code class="language-plaintext highlighter-rouge">parseExpr</code> to parse
the regular textual representation into a NimNode. For example
<code class="language-plaintext highlighter-rouge">parseStmt("result = 10")</code> returns this AST:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>StmtList
Asgn
Ident !"result"
IntLit 10
</code></pre></div></div>
<p>A very useful way to find the AST of a piece of code is <code class="language-plaintext highlighter-rouge">dumpTree</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">macros</span>
<span class="n">dumpTree</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="mi">10</span></code></pre></figure>
<p>This is the same output as you get with <code class="language-plaintext highlighter-rouge">treeRepr</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">macros</span>
<span class="k">static</span><span class="p">:</span>
<span class="n">echo</span> <span class="n">treeRepr</span><span class="p">(</span><span class="n">parseStmt</span><span class="p">(</span><span class="s">"result = 10"</span><span class="p">))</span></code></pre></figure>
<p>Alternatively you can use <code class="language-plaintext highlighter-rouge">lispRepr</code> to get a lisp-like representation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>StmtList(Asgn(Ident(!"result"), IntLit(10)))
</code></pre></div></div>
<p>Finally there is also the <code class="language-plaintext highlighter-rouge">repr</code> proc, which turns a NimNode AST back into its
textual representation.</p>
<p>Many beginners start by piecing strings together and finally calling
<code class="language-plaintext highlighter-rouge">parseStmt</code> on them. While this works it is inefficient and prone to bugs.
Instead you can use the <a href="http://nim-lang.org/docs/macros.html">macros module</a>
to create NimNodes of all kinds yourself. <code class="language-plaintext highlighter-rouge">dumpTree</code> gives you a hint if you’re
not sure how a specific piece of code will look in its AST representation.</p>
<h3 id="json-parsing">JSON Parsing</h3>
<p>JSON is pretty popular, so let’s improve the support for it in Nim. What we
want is to have a magical <code class="language-plaintext highlighter-rouge">%*</code> so that we can write JSON directly in Nim source
code and have it checked at compile time, like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">j1</span> <span class="o">=</span> <span class="o">%*</span>
<span class="o">[</span>
<span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="s">"John"</span><span class="p">,</span>
<span class="s">"age"</span><span class="p">:</span> <span class="mi">30</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="s">"Susan"</span><span class="p">,</span>
<span class="s">"age"</span><span class="p">:</span> <span class="mi">31</span>
<span class="p">}</span>
<span class="o">]</span></code></pre></figure>
<p>So far if you want to use JSON in Nim, you have to use the JSON constructor <code class="language-plaintext highlighter-rouge">%</code>
a lot:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">json</span>
<span class="k">var</span> <span class="n">j2</span> <span class="o">=</span>
<span class="o">%[</span>
<span class="o">%</span><span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="o">%</span><span class="s">"John"</span><span class="p">,</span>
<span class="s">"age"</span><span class="p">:</span> <span class="o">%</span><span class="mi">30</span>
<span class="p">},</span>
<span class="o">%</span><span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="o">%</span><span class="s">"Susan"</span><span class="p">,</span>
<span class="s">"age"</span><span class="p">:</span> <span class="o">%</span><span class="mi">31</span>
<span class="p">}</span>
<span class="o">]</span></code></pre></figure>
<p>Looks annoying. How can we implement <code class="language-plaintext highlighter-rouge">%*</code>? As a macro of course!:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">macro</span> <span class="p">`</span><span class="o">%*</span><span class="p">`</span><span class="o">*</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">expr</span><span class="p">):</span> <span class="n">expr</span> <span class="o">=</span> <span class="n">toJson</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></code></pre></figure>
<p>Ok, that doesn’t do anything interesting yet. We just call the still
unspecified compile time proc <code class="language-plaintext highlighter-rouge">toJson</code> and return the result. We want <code class="language-plaintext highlighter-rouge">toJson</code>
to traverse the passed AST <code class="language-plaintext highlighter-rouge">x</code> and create a new AST, which inserts a <code class="language-plaintext highlighter-rouge">%</code> call
at just the right places, exactly as it would happen if we added the <code class="language-plaintext highlighter-rouge">%</code> calls
manually.</p>
<p>For this purpose we print the AST of <code class="language-plaintext highlighter-rouge">j2</code> by putting it into <code class="language-plaintext highlighter-rouge">dumpTree</code> from
the <a href="http://nim-lang.org/docs/macros.html">macros module</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">json</span><span class="p">,</span> <span class="n">macros</span>
<span class="n">dumpTree</span><span class="p">:</span>
<span class="o">%[</span>
<span class="o">%</span><span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="o">%</span><span class="s">"John"</span><span class="p">,</span>
<span class="s">"age"</span><span class="p">:</span> <span class="o">%</span><span class="mi">30</span>
<span class="p">},</span>
<span class="o">%</span><span class="p">{</span>
<span class="s">"name"</span><span class="p">:</span> <span class="o">%</span><span class="s">"Susan"</span><span class="p">,</span>
<span class="s">"age"</span><span class="p">:</span> <span class="o">%</span><span class="mi">31</span>
<span class="p">}</span>
<span class="o">]</span></code></pre></figure>
<p>We get the following AST printed when compiling this program:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Prefix
Ident !"%"
Bracket
Prefix
Ident !"%"
TableConstr
ExprColonExpr
StrLit name
Prefix
Ident !"%"
StrLit John
ExprColonExpr
StrLit age
Prefix
Ident !"%"
IntLit 30
Prefix
Ident !"%"
TableConstr
ExprColonExpr
StrLit name
Prefix
Ident !"%"
StrLit Susan
ExprColonExpr
StrLit age
Prefix
Ident !"%"
IntLit 31
</code></pre></div></div>
<p>This turned out quite big, but from here we can see how the AST we want to
construct looks like. We do the same for <code class="language-plaintext highlighter-rouge">j1</code> to see what we’re working with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>StmtList
Bracket
TableConstr
ExprColonExpr
StrLit name
StrLit John
ExprColonExpr
StrLit age
IntLit 30
TableConstr
ExprColonExpr
StrLit name
StrLit Susan
ExprColonExpr
StrLit age
IntLit 31
</code></pre></div></div>
<p>The idea now is to insert a <code class="language-plaintext highlighter-rouge">%</code> at each level, except in front of the <code class="language-plaintext highlighter-rouge">"name"</code>
and <code class="language-plaintext highlighter-rouge">"age"</code> in our case, the first elements in colon expressions.</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">toJson</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">PNimrodNode</span><span class="p">):</span> <span class="n">PNimrodNode</span> <span class="p">{.</span><span class="n">compiletime</span><span class="p">.}</span> <span class="o">=</span>
<span class="k">case</span> <span class="n">x</span><span class="p">.</span><span class="n">kind</span>
<span class="k">of</span> <span class="n">nnkBracket</span><span class="p">:</span> <span class="c"># Corresponds to Bracket in dumpTree</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newNimNode</span><span class="p">(</span><span class="n">nnkBracket</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">0</span> <span class="p">..</span> <span class="o"><</span><span class="n">x</span><span class="p">.</span><span class="n">len</span><span class="p">:</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">toJson</span><span class="p">(</span><span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">))</span> <span class="c"># Recurse to add %</span>
<span class="k">of</span> <span class="n">nnkTableConstr</span><span class="p">:</span> <span class="c"># nnk stands for Nim node kind</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newNimNode</span><span class="p">(</span><span class="n">nnkTableConstr</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">0</span> <span class="p">..</span> <span class="o"><</span><span class="n">x</span><span class="p">.</span><span class="n">len</span><span class="p">:</span>
<span class="n">assert</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">.</span><span class="n">kind</span> <span class="o">==</span> <span class="n">nnkExprColonExpr</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">newNimNode</span><span class="p">(</span><span class="n">nnkExprColonExpr</span><span class="p">)</span>
<span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">][</span><span class="mi">0</span><span class="o">]</span><span class="p">)</span> <span class="c"># First element: no %</span>
<span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">toJson</span><span class="p">(</span><span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">][</span><span class="mi">1</span><span class="o">]</span><span class="p">))):</span> <span class="c"># Second element: Recurse to add %</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">x</span> <span class="c"># End of recursion</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">result</span><span class="p">.</span><span class="n">prefix</span><span class="p">(</span><span class="s">"%"</span><span class="p">)</span> <span class="c"># Surround this level with %</span></code></pre></figure>
<p>And that’s it! Now our <code class="language-plaintext highlighter-rouge">%*</code> works just as we want it to. If we did anything
wrong, we can modify the macro to check the actual code it produces:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">macro</span> <span class="p">`</span><span class="o">%*</span><span class="p">`</span><span class="o">*</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">expr</span><span class="p">):</span> <span class="n">expr</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">toJson</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">echo</span> <span class="n">result</span><span class="p">.</span><span class="n">repr</span> <span class="c"># Print code representation of AST</span></code></pre></figure>
<p>This prints:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% [% {"name": % "John", "age": % 30}, % {"name": % "Susan", "age": % 31}]
</code></pre></div></div>
<p>Perfect! This macro we just developed landed in Nim’s <a href="http://nim-lang.org/docs/json.html">json
module</a> already.</p>
<h3 id="enum-parsing-optimization">Enum Parsing optimization</h3>
<p>With enums we can create new types that contain ordered values, just like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"> <span class="k">type</span> <span class="n">Fruit</span> <span class="o">=</span> <span class="k">enum</span> <span class="n">Apple</span><span class="p">,</span> <span class="n">Banana</span><span class="p">,</span> <span class="n">Cherry</span></code></pre></figure>
<p>Strings can be parsed to an enum using <code class="language-plaintext highlighter-rouge">parseEnum</code> from <a href="http://nim-lang.org/docs/strutils.html">strutils</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"> <span class="k">let</span> <span class="n">fruit</span> <span class="o">=</span> <span class="n">parseEnum</span><span class="o">[</span><span class="n">Fruit</span><span class="o">]</span><span class="p">(</span><span class="s">"cherry"</span><span class="p">)</span></code></pre></figure>
<p>If we do this a lot, we notice that it’s kind of slow though:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">1</span> <span class="p">..</span> <span class="mi">10_000_000</span><span class="p">:</span>
<span class="k">var</span> <span class="n">select</span> <span class="o">=</span> <span class="n">parseEnum</span><span class="o">[</span><span class="n">Fruit</span><span class="o">]</span><span class="p">(</span><span class="s">"cherry"</span><span class="p">)</span>
<span class="n">doAssert</span> <span class="n">select</span> <span class="o">==</span> <span class="n">Cherry</span></code></pre></figure>
<p>This takes 2.2 seconds on my machine. Let’s look at the definition of
<code class="language-plaintext highlighter-rouge">parseEnum</code> to find out why:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">parseEnum</span><span class="o">*[</span><span class="n">T</span><span class="p">:</span> <span class="k">enum</span><span class="o">]</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="n">T</span> <span class="o">=</span>
<span class="sd">## Parses an enum ``T``.</span>
<span class="sd">##</span>
<span class="sd">## Raises ``ValueError`` for an invalid value in `s`. The</span>
<span class="sd">## comparison is done in a style insensitive way.</span>
<span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">low</span><span class="p">(</span><span class="n">T</span><span class="p">)..</span><span class="n">high</span><span class="p">(</span><span class="n">T</span><span class="p">):</span>
<span class="k">if</span> <span class="n">cmpIgnoreStyle</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="o">$</span><span class="n">e</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="n">e</span>
<span class="k">raise</span> <span class="n">newException</span><span class="p">(</span><span class="n">ValueError</span><span class="p">,</span> <span class="s">"invalid enum value: "</span> <span class="o">&</span> <span class="n">s</span><span class="p">)</span></code></pre></figure>
<p>We can see the problem already. We iterate through all the values inside the
enum type, from <code class="language-plaintext highlighter-rouge">low(T)</code> to <code class="language-plaintext highlighter-rouge">high(T)</code>. Then <code class="language-plaintext highlighter-rouge">$e</code> creates a string of each enum
value, which is quite expensive. Since we already know the type of the enum at
compile time, we could create the strings at compile time as well.</p>
<p>Again, let’s think about what we want the result to look like before writing
the macro. Basically what we want to do is unroll the for loop at compile time:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">if</span> <span class="n">cmpIgnoreStyle</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="s">"Apple"</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="n">Apple</span>
<span class="k">if</span> <span class="n">cmpIgnoreStyle</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="s">"Banana"</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="n">Banana</span>
<span class="k">if</span> <span class="n">cmpIgnoreStyle</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="s">"Cherry"</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="n">Cherry</span>
<span class="k">raise</span> <span class="n">newException</span><span class="p">(</span><span class="n">ValueError</span><span class="p">,</span> <span class="s">"invalid enum value: "</span> <span class="o">&</span> <span class="n">s</span><span class="p">)</span></code></pre></figure>
<p>Now we can create the proc. Other than in the last example we won’t create the
AST manually this time. Instead we use <code class="language-plaintext highlighter-rouge">parseStmt</code> to create a statement AST
from a string containing Nim code. An equivalent <code class="language-plaintext highlighter-rouge">parseExpr</code> for expressions
exists as well. Here’s how the final proc with a macro inside looks:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">parseEnum</span><span class="o">*[</span><span class="n">T</span><span class="p">:</span> <span class="k">enum</span><span class="o">]</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="n">T</span> <span class="o">=</span>
<span class="k">macro</span> <span class="n">m</span><span class="p">:</span> <span class="n">untyped</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newStmtList</span><span class="p">()</span>
<span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">T</span><span class="p">:</span> <span class="n">result</span><span class="p">.</span><span class="n">add</span> <span class="n">parseStmt</span><span class="p">(</span>
<span class="s">"if cmpIgnoreStyle(s, </span><span class="se">\"</span><span class="si">$1</span><span class="se">\"</span><span class="s">) == 0: return </span><span class="si">$1</span><span class="s">"</span><span class="p">.</span><span class="n">format</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span> <span class="n">parseStmt</span><span class="p">(</span>
<span class="s">"raise newException(ValueError, </span><span class="se">\"</span><span class="s">invalid enum value: </span><span class="se">\"</span><span class="s">&s)"</span><span class="p">)</span>
<span class="c">#echo result.repr # To make sure we get what we want</span>
<span class="n">m</span><span class="p">()</span> <span class="c"># Actually invoke the macro to insert the statements here</span></code></pre></figure>
<p>Running the same code with our new implementation of parseEnum takes 0.5
seconds now, about 4 times faster than before. Great!</p>
<h3 id="html-dsl">HTML DSL</h3>
<p>We can use Nim’s templates and macros to create domain specific languages
(DSL) that are translated into Nim code at compiletime. Nim’s syntax is quite
flexible, so this is a powerful tool. As an example we build a simple HTML DSL.</p>
<p>The goal is to be able to write this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">page</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">content</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{.</span><span class="n">htmlTemplate</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">html</span><span class="p">:</span>
<span class="n">head</span><span class="p">:</span>
<span class="n">title</span><span class="p">:</span> <span class="n">title</span>
<span class="n">body</span><span class="p">:</span>
<span class="n">h1</span><span class="p">:</span> <span class="n">title</span>
<span class="n">p</span><span class="p">:</span> <span class="s">"Default Content"</span>
<span class="n">p</span><span class="p">:</span> <span class="n">content</span>
<span class="n">echo</span> <span class="n">page</span><span class="p">(</span><span class="s">"My own website"</span><span class="p">,</span> <span class="s">"My extra content"</span><span class="p">)</span></code></pre></figure>
<p>And thus print the following HTML:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>
My own website
<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><h1></span>
My own website
<span class="nt"></h1></span>
<span class="nt"><p></span>
Default Content
<span class="nt"></p></span>
<span class="nt"><p></span>
My extra content
<span class="nt"></p></span>
<span class="nt"></body></span>
<span class="nt"></html></span></code></pre></figure>
<p>For convenience we want to use the <code class="language-plaintext highlighter-rouge">htmlTemplate</code> macro as a pragma, annotated
as <code class="language-plaintext highlighter-rouge">{.htmlTemplate.}</code>. Instead we could also write it in this way:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="n">htmlTemplate</span><span class="p">:</span>
<span class="k">proc </span><span class="nf">page</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">content</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="n">html</span><span class="p">:</span>
<span class="n">head</span><span class="p">:</span>
<span class="n">title</span><span class="p">:</span> <span class="n">title</span>
<span class="n">body</span><span class="p">:</span>
<span class="n">h1</span><span class="p">:</span> <span class="n">title</span>
<span class="n">p</span><span class="p">:</span> <span class="s">"Default Content"</span>
<span class="n">p</span><span class="p">:</span> <span class="n">content</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">htmlTemplate</code> macro shall transform the <code class="language-plaintext highlighter-rouge">page</code> proc, adding a <code class="language-plaintext highlighter-rouge">string</code>
return type and creating a new body out of the DSL definition, into this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">page</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">content</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span> <span class="s">"<html></span><span class="se">\n</span><span class="s">"</span>
<span class="p">...</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span> <span class="s">"</html></span><span class="se">\n</span><span class="s">"</span></code></pre></figure>
<p>Looks simple enough, here’s how the macro works:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">macro</span> <span class="n">htmlTemplate</span><span class="p">(</span><span class="n">procDef</span><span class="p">:</span> <span class="n">expr</span><span class="p">):</span> <span class="n">untyped</span> <span class="o">=</span>
<span class="n">procDef</span><span class="p">.</span><span class="n">expectKind</span> <span class="n">nnkProcDef</span>
<span class="c"># Same name as specified</span>
<span class="k">let</span> <span class="n">name</span> <span class="o">=</span> <span class="n">procDef</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span>
<span class="c"># Return type: string</span>
<span class="k">var</span> <span class="n">params</span> <span class="o">=</span> <span class="o">@[</span><span class="n">newIdentNode</span><span class="p">(</span><span class="s">"string"</span><span class="p">)</span><span class="o">]</span>
<span class="c"># Same parameters as specified</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="o"><</span><span class="n">procDef</span><span class="o">[</span><span class="mi">3</span><span class="o">]</span><span class="p">.</span><span class="n">len</span><span class="p">:</span>
<span class="n">params</span><span class="p">.</span><span class="n">add</span> <span class="n">procDef</span><span class="o">[</span><span class="mi">3</span><span class="o">][</span><span class="n">i</span><span class="o">]</span>
<span class="k">var</span> <span class="n">body</span> <span class="o">=</span> <span class="n">newStmtList</span><span class="p">()</span>
<span class="c"># result = ""</span>
<span class="n">body</span><span class="p">.</span><span class="n">add</span> <span class="n">newAssignment</span><span class="p">(</span><span class="n">newIdentNode</span><span class="p">(</span><span class="s">"result"</span><span class="p">),</span>
<span class="n">newStrLitNode</span><span class="p">(</span><span class="s">""</span><span class="p">))</span>
<span class="c"># Recurse over DSL definition</span>
<span class="n">body</span><span class="p">.</span><span class="n">add</span> <span class="n">htmlInner</span><span class="p">(</span><span class="n">procDef</span><span class="o">[</span><span class="mi">6</span><span class="o">]</span><span class="p">)</span>
<span class="c"># Return a new proc</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newStmtList</span><span class="p">(</span><span class="n">newProc</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">params</span><span class="p">,</span> <span class="n">body</span><span class="p">))</span></code></pre></figure>
<p>The real magic of recursively handling the HTML tags happens in <code class="language-plaintext highlighter-rouge">htmlInner</code> of
course, a compiletime proc that calls itself recursively to iterate over the
body definition:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">write</span><span class="p">(</span><span class="n">arg</span><span class="p">:</span> <span class="n">expr</span><span class="p">)</span> <span class="o">=</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span> <span class="n">newCall</span><span class="p">(</span><span class="s">"add"</span><span class="p">,</span> <span class="n">newIdentNode</span><span class="p">(</span><span class="s">"result"</span><span class="p">),</span> <span class="n">arg</span><span class="p">)</span>
<span class="k">template</span> <span class="n">writeLit</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="p">`</span><span class="o">$</span><span class="p">`</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="n">write</span> <span class="n">newStrLitNode</span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">join</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">htmlInner</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">NimNode</span><span class="p">,</span> <span class="n">indent</span> <span class="o">=</span> <span class="mi">0</span><span class="p">):</span> <span class="n">NimNode</span>
<span class="p">{.</span><span class="n">compiletime</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">x</span><span class="p">.</span><span class="n">expectKind</span> <span class="n">nnkStmtList</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newStmtList</span><span class="p">()</span>
<span class="k">let</span> <span class="n">spaces</span> <span class="o">=</span> <span class="n">repeat</span><span class="p">(</span><span class="sc">' '</span><span class="p">,</span> <span class="n">indent</span><span class="p">)</span>
<span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="n">x</span><span class="p">:</span>
<span class="k">case</span> <span class="n">y</span><span class="p">.</span><span class="n">kind</span>
<span class="k">of</span> <span class="n">nnkCall</span><span class="p">:</span>
<span class="n">y</span><span class="p">.</span><span class="n">expectLen</span> <span class="mi">2</span>
<span class="k">let</span> <span class="n">tag</span> <span class="o">=</span> <span class="n">y</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span>
<span class="n">tag</span><span class="p">.</span><span class="n">expectKind</span> <span class="n">nnkIdent</span>
<span class="n">writeLit</span> <span class="n">spaces</span><span class="p">,</span> <span class="s">"<"</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="s">"></span><span class="se">\n</span><span class="s">"</span>
<span class="c"># Recurse over child</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span> <span class="n">htmlInner</span><span class="p">(</span><span class="n">y</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span><span class="p">,</span> <span class="n">indent</span> <span class="o">+</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">writeLit</span> <span class="n">spaces</span><span class="p">,</span> <span class="s">"</"</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="s">"></span><span class="se">\n</span><span class="s">"</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">writeLit</span> <span class="n">spaces</span>
<span class="n">write</span> <span class="n">y</span>
<span class="n">writeLit</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span></code></pre></figure>
<p>We can check that we get the expected output by adding a simple <code class="language-plaintext highlighter-rouge">echo
result.repr</code> at the end of <code class="language-plaintext highlighter-rouge">htmlTemplate</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">page</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">content</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">add</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="s">"<html></span><span class="se">\x0A</span><span class="s">"</span><span class="p">)</span>
<span class="n">add</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="s">" <head></span><span class="se">\x0A</span><span class="s">"</span><span class="p">)</span>
<span class="p">...</span>
<span class="n">add</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="s">"</html></span><span class="se">\x0A</span><span class="s">"</span><span class="p">)</span></code></pre></figure>
<p>Where <code class="language-plaintext highlighter-rouge">\x0A</code> is just the newline character. Looks good and the output works!</p>
<p><a href="http://flyx.github.io/emerald/">emerald</a> is a much more complete HTML DSL that
works in a similar manner. A simpler HTML generator is included in the
standard library in the <a href="http://nim-lang.org/docs/htmlgen.html">htmlgen</a>
module.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope you enjoyed this trip through Nim’s metaprogramming capabilities. Always
remember: With great power comes great responsibility, so use the least
powerful construct that does the job. This reduces complexity and makes it
easier to understand the code and keep it maintainable.</p>
<p>For further information and reference see:</p>
<ul>
<li><a href="http://nim-lang.org/docs/tut2.html#templates">Nim Tutorial (Part 2)</a></li>
<li><a href="http://nim-lang.org/docs/manual.html#templates">Nim Manual</a></li>
<li><a href="http://nim-lang.org/docs/macros.html">Macros Module</a></li>
</ul>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=11851234">Hacker News</a> and <a href="https://www.reddit.com/r/programming/comments/4mpuh7/introduction_to_metaprogramming_in_nim/">r/programming</a>.</p>
DDNet Server Statistics with ServerStatus, RRDtool and Nim2016-05-14T00:00:00+02:00https://hookrace.net/blog/server-statistics
<p>About a month ago I set up <a href="https://ddnet.org/stats/server/">statistics</a> for the official <a href="https://ddnet.org/">DDNet</a> servers. My motivations for this are:</p>
<ol>
<li>Monitor the servers more easily</li>
<li>Get notified about server problems</li>
<li>Have nice graphs to look at</li>
</ol>
<p>The choices for the software used are mainly made to keep resource usage low, a
general principle used for DDNet since we run on cheap VPSes all around the
world and are limited in CPU and memory resources. In the rest of this post we
will explore the 3 major tools used, their purpose in our solution as well as
their performance impact:</p>
<ol>
<li><a href="https://github.com/BotoX/ServerStatus">ServerStatus</a>: Gather live server statistics</li>
<li><a href="https://oss.oetiker.ch/rrdtool/index.en.html">RRDtool</a>: Record and graph data</li>
<li><a href="http://nim-lang.org/">Nim</a>: Favorite programming language for performance and readability</li>
</ol>
<!--more-->
<h2 id="gathering-live-server-statistics-with-serverstatus">Gathering live server statistics with ServerStatus</h2>
<p>We’ve been running BotoX’s <a href="https://github.com/BotoX/ServerStatus">ServerStatus</a> to get <a href="https://ddnet.org/status/">live server statistics</a> for some time now. It works quite well to quickly notice major server problems like a high load or incoming (D)DoS attacks, provided that you keep an eye out for it.</p>
<p>We use ServerStatus by running its <a href="https://github.com/BotoX/ServerStatus/blob/master/clients/client.py">simple Python client</a> on each server to gather interesting information. The client transmits that data by TCP to the C/C++ server, which aggregates it into <a href="https://ddnet.org/status/json/stats.json">a JSON file</a>. This JSON file is then fetched and displayed every two seconds by the JavaScript frontend of <a href="https://ddnet.org/status/">our Status page</a>.</p>
<p>On a regular Saturday morning the end result looks like this:</p>
<p><img src="/public/ddnet-status.png" alt="DDNet Status" /></p>
<p>On a quick glance you notice that the DDNet.tw server has high CPU usage (which is totally normal since it runs some hefty cron jobs every 20 minutes) and DDNet RUS is receiving a small DoS attack with just 1.6 MB/s (unfortunately also totally normal). Apart from that everything looks fine.</p>
<p>ServerStatus footprint, calculated from Linux <code class="language-plaintext highlighter-rouge">/proc</code> statistics as follows (in Nim):</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">os</span><span class="p">,</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">posix</span><span class="p">,</span> <span class="n">strfmt</span>
<span class="c"># CPU and memory usage of process, based on PROC(5) and</span>
<span class="c"># http://stackoverflow.com/a/16736599</span>
<span class="k">let</span>
<span class="n">pid</span> <span class="o">=</span> <span class="n">paramStr</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c"># Clock ticks per second</span>
<span class="n">frequency</span> <span class="o">=</span> <span class="n">sysconf</span><span class="p">(</span><span class="n">SC_CLK_TCK</span><span class="p">)</span>
<span class="c"># Size of memory page in bytes</span>
<span class="n">pagesize</span> <span class="o">=</span> <span class="n">sysconf</span><span class="p">(</span><span class="n">SC_PAGESIZE</span><span class="p">)</span>
<span class="n">uptime</span> <span class="o">=</span> <span class="n">readFile</span><span class="p">(</span><span class="s">"/proc/uptime"</span><span class="p">).</span><span class="n">split</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">.</span><span class="n">parseFloat</span>
<span class="n">fields</span> <span class="o">=</span> <span class="n">readFile</span><span class="p">(</span><span class="s">"/proc/"</span> <span class="o">&</span> <span class="n">pid</span> <span class="o">&</span> <span class="s">"/stat"</span><span class="p">).</span><span class="n">split</span>
<span class="c"># Amount of time in user mode</span>
<span class="n">utime</span> <span class="o">=</span> <span class="n">fields</span><span class="o">[</span><span class="mi">13</span><span class="o">]</span><span class="p">.</span><span class="n">parseInt</span>
<span class="c"># Amount of time in kernel mode</span>
<span class="n">stime</span> <span class="o">=</span> <span class="n">fields</span><span class="o">[</span><span class="mi">14</span><span class="o">]</span><span class="p">.</span><span class="n">parseInt</span>
<span class="c"># Amount of children time in user mode</span>
<span class="n">cutime</span> <span class="o">=</span> <span class="n">fields</span><span class="o">[</span><span class="mi">15</span><span class="o">]</span><span class="p">.</span><span class="n">parseInt</span>
<span class="c"># Amount of children time in kernel mode</span>
<span class="n">cstime</span> <span class="o">=</span> <span class="n">fields</span><span class="o">[</span><span class="mi">16</span><span class="o">]</span><span class="p">.</span><span class="n">parseInt</span>
<span class="c"># Time process started after boot</span>
<span class="n">starttime</span> <span class="o">=</span> <span class="n">fields</span><span class="o">[</span><span class="mi">21</span><span class="o">]</span><span class="p">.</span><span class="n">parseInt</span>
<span class="c"># Resident Set Size: number of pages in memory</span>
<span class="n">rssmem</span> <span class="o">=</span> <span class="n">fields</span><span class="o">[</span><span class="mi">23</span><span class="o">]</span><span class="p">.</span><span class="n">parseInt</span>
<span class="n">totaltime</span> <span class="o">=</span> <span class="n">utime</span> <span class="o">+</span> <span class="n">stime</span> <span class="o">+</span> <span class="n">cutime</span> <span class="o">+</span> <span class="n">cstime</span>
<span class="n">seconds</span> <span class="o">=</span> <span class="n">uptime</span> <span class="o">-</span> <span class="p">(</span><span class="n">starttime</span> <span class="o">/</span> <span class="n">frequency</span><span class="p">)</span>
<span class="n">cpuusage</span> <span class="o">=</span> <span class="mi">100</span> <span class="o">*</span> <span class="p">(</span><span class="n">totaltime</span> <span class="o">/</span> <span class="n">frequency</span><span class="p">)</span> <span class="o">/</span> <span class="n">seconds</span>
<span class="n">memusage</span> <span class="o">=</span> <span class="n">rssmem</span> <span class="o">*</span> <span class="n">pagesize</span> <span class="o">/</span> <span class="mi">1_000_000</span>
<span class="n">echo</span> <span class="s">interp"${cpuusage:.2f} % CPU ${memusage:.2f} MB Memory"</span></code></pre></figure>
<table>
<thead>
<tr>
<th>Part</th>
<th style="text-align: right">CPU</th>
<th style="text-align: right">Memory</th>
</tr>
</thead>
<tbody>
<tr>
<td>Client</td>
<td style="text-align: right">0.14 %</td>
<td style="text-align: right">3.70 MB</td>
</tr>
<tr>
<td>Server (9 clients)</td>
<td style="text-align: right">0.58 %</td>
<td style="text-align: right">0.18 MB</td>
</tr>
</tbody>
</table>
<h2 id="recording-and-graphing-data-with-rrdtool">Recording and graphing data with RRDtool</h2>
<p>I haven’t used RRDtool for about 7 years, but it’s still an excellent tool to record data into a fixed-size round robin database. For us three functions of RRDtool are important: <code class="language-plaintext highlighter-rouge">create</code> to create the database, <code class="language-plaintext highlighter-rouge">update</code> to add a new value to be aggregated into the database, and <code class="language-plaintext highlighter-rouge">graph</code> to render the database into a beautiful graph.</p>
<p>CPU, network and memory are the most important resources for me, so their usage should be recorded. Let us use network traffic as an example and create a database:</p>
<p>First we need to think about what data we want to record in the RRD:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1 sample = 30 seconds
1 day = 2880 samples = 6 * 480 pixels, each pixel is 03:00 min
7 days = 20160 samples = 42 * 480 pixels, each pixel is 21:00 min
49 days = 141120 samples = 147 * 960 pixels, each pixel is 73:30 min
</code></pre></div></div>
<p>Then we can use this to create the actual database file:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rrdtool create ddnet.tw-net.rrd <span class="sb">`</span><span class="c"># File name` \</span>
<span class="nt">--step</span> 30 <span class="sb">`</span><span class="c"># Interval in seconds with which data is fed` \</span>
DS:network_rx:GAUGE:60:0:U <span class="sb">`</span><span class="c"># Data source receiving` \</span>
DS:network_tx:GAUGE:60:0:U <span class="sb">`</span><span class="c"># DS Sending` \</span>
RRA:AVERAGE:0.5:6:480 <span class="sb">`</span><span class="c"># Round robin archive for 1 day` \</span>
RRA:AVERAGE:0.5:42:480 <span class="sb">`</span><span class="c"># RRA for 7 days` \</span>
RRA:AVERAGE:0.5:147:960 <span class="sb">`</span><span class="c"># RRA for 49 days`</span></code></pre></figure>
<p>If you’re curious about what exactly happens here, you can find more information in <a href="https://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html">rrdcreate(1)</a>.</p>
<p>The resulting ddnet.tw-net.rrd file is just 32 KB in size and will forever stay that exact size. (All our databases together are just 1 MB.) New data in each round robin archive simply overwrites the oldest data. A disadvantage of RRDtool is that you need to think ahead and plan what data you want to store.</p>
<p>The next step is to put new data into our little database, which we should do every 30 seconds:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rrdtool update ddnet.tw-net.rrd N:42:1234</code></pre></figure>
<p>Super simple! <code class="language-plaintext highlighter-rouge">42</code> is our <code class="language-plaintext highlighter-rouge">network_rx</code> value, <code class="language-plaintext highlighter-rouge">1234</code> the value for <code class="language-plaintext highlighter-rouge">network_tx</code>. These values are now aggregated using the <code class="language-plaintext highlighter-rouge">AVERAGE</code> and finally put into their respective archives.</p>
<p>Once we have enough values we can finally create the graph, for example for 1 day:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rrdtool graph ddnet.tw-net-1d.png <span class="nt">--rigid</span> <span class="nt">--base</span> 1000 <span class="se">\</span>
<span class="nt">--width</span> 419 <span class="nt">--height</span> 150 <span class="nt">--logarithmic</span> <span class="nt">--units</span><span class="o">=</span>si <span class="nt">-a</span> PNG <span class="se">\</span>
<span class="sb">`</span><span class="c"># Calculation over last day only` \</span>
<span class="nt">--vertical-label</span> <span class="s2">"Bytes/s"</span> <span class="nt">--start</span> now-1d <span class="se">\</span>
<span class="sb">`</span><span class="c"># Fetch data from RRD file` \</span>
DEF:network_rx<span class="o">=</span>ddnet.tw-net.rrd:network_rx:AVERAGE <span class="se">\</span>
DEF:network_tx<span class="o">=</span>ddnet.tw-net.rrd:network_tx:AVERAGE <span class="se">\</span>
<span class="sb">`</span><span class="c"># Calculate aggregates based on data` \</span>
VDEF:network_rx_a<span class="o">=</span>network_rx,AVERAGE <span class="se">\</span>
VDEF:network_rx_m<span class="o">=</span>network_rx,MAXIMUM <span class="se">\</span>
VDEF:network_rx_c<span class="o">=</span>network_rx,LAST <span class="se">\</span>
VDEF:network_rx_s<span class="o">=</span>network_rx,TOTAL <span class="se">\</span>
VDEF:network_tx_a<span class="o">=</span>network_tx,AVERAGE <span class="se">\</span>
VDEF:network_tx_m<span class="o">=</span>network_tx,MAXIMUM <span class="se">\</span>
VDEF:network_tx_c<span class="o">=</span>network_tx,LAST <span class="se">\</span>
VDEF:network_tx_s<span class="o">=</span>network_tx,TOTAL <span class="se">\</span>
<span class="sb">`</span><span class="c"># Draw area graph in light colors` \</span>
AREA:network_tx#fee8c8: <span class="se">\</span>
AREA:network_rx#e0e0e0: <span class="se">\</span>
<span class="sb">`</span><span class="c"># Draw clear area outline on top` \</span>
LINE1:network_tx#e34a33:<span class="s2">"out"</span> <span class="se">\</span>
<span class="sb">`</span><span class="c"># Print aggregate values to legend` \</span>
GPRINT:network_tx_a:<span class="s2">"avg</span><span class="se">\:</span><span class="s2"> %6.2lf %sB"</span> <span class="se">\</span>
GPRINT:network_tx_m:<span class="s2">"max</span><span class="se">\:</span><span class="s2"> %6.2lf %sB"</span> <span class="se">\</span>
GPRINT:network_tx_c:<span class="s2">"cur</span><span class="se">\:</span><span class="s2"> %6.2lf %sB"</span> <span class="se">\</span>
GPRINT:network_tx_s:<span class="s2">"sum</span><span class="se">\:</span><span class="s2"> %6.2lf %sB</span><span class="se">\n</span><span class="s2">"</span> <span class="se">\</span>
<span class="sb">`</span><span class="c"># Other area outline` \</span>
LINE1:network_rx#636363:<span class="s2">"in "</span> <span class="se">\</span>
GPRINT:network_rx_a:<span class="s2">"avg</span><span class="se">\:</span><span class="s2"> %6.2lf %sB"</span> <span class="se">\</span>
GPRINT:network_rx_m:<span class="s2">"max</span><span class="se">\:</span><span class="s2"> %6.2lf %sB"</span> <span class="se">\</span>
GPRINT:network_rx_c:<span class="s2">"cur</span><span class="se">\:</span><span class="s2"> %6.2lf %sB"</span> <span class="se">\</span>
GPRINT:network_rx_s:<span class="s2">"sum</span><span class="se">\:</span><span class="s2"> %6.2lf %sB</span><span class="se">\n</span><span class="s2">"</span></code></pre></figure>
<p>As always, the <a href="http://oss.oetiker.ch/rrdtool/doc/rrdgraph.en.html">manual of rrdgraph</a> explains the possibilities.</p>
<p>I’m using RRDtool 1.6.0 instead of 1.4.8 because I very much prefer its density of x-axis labels. Here are the outputs of our database:</p>
<p>RRDtool 1.4.8: <img src="/public/ger.ddnet.tw-net-1d-1.4.8.png" alt="RRDtool 1.4.8" />
RRDtool 1.6.0: <img src="/public/ger.ddnet.tw-net-1d-1.6.0.png" alt="RRDtool 1.6.0" /></p>
<p>RRDtool footprint:</p>
<table>
<thead>
<tr>
<th>Part</th>
<th style="text-align: right">Runtime</th>
<th style="text-align: right">Memory</th>
</tr>
</thead>
<tbody>
<tr>
<td>Create (9 servers)</td>
<td style="text-align: right">0.01 s</td>
<td style="text-align: right">1.74 MB</td>
</tr>
<tr>
<td>Graph (9 servers)</td>
<td style="text-align: right">1.99 s</td>
<td style="text-align: right">2.82 MB</td>
</tr>
</tbody>
</table>
<h2 id="putting-it-together-with-nim">Putting it together with Nim</h2>
<p>To aggregate the raw data from ServerStatus into 30-second packets I use a small Nim program. It automatically creates new databases when a new server is added and keeps them updated:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">common</span><span class="p">,</span> <span class="n">json</span><span class="p">,</span> <span class="n">osproc</span><span class="p">,</span> <span class="n">os</span><span class="p">,</span> <span class="n">times</span><span class="p">,</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">tables</span>
<span class="k">type</span>
<span class="n">Data</span> <span class="o">=</span> <span class="k">object</span>
<span class="n">network_rx</span><span class="p">,</span> <span class="n">network_tx</span><span class="p">:</span> <span class="n">BiggestInt</span>
<span class="n">cpu</span><span class="p">,</span> <span class="n">memory_used</span><span class="p">,</span> <span class="n">memory_total</span><span class="p">,</span> <span class="n">swap_used</span><span class="p">,</span> <span class="n">swap_total</span><span class="p">:</span> <span class="n">BiggestInt</span>
<span class="n">load</span><span class="p">:</span> <span class="kt">float</span>
<span class="k">const</span> <span class="n">freq</span> <span class="o">=</span> <span class="mi">30</span> <span class="c"># report new data to rrd every 30 seconds</span>
<span class="k">var</span>
<span class="n">lastUpdated</span><span class="p">:</span> <span class="n">BiggestInt</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">dataTable</span> <span class="o">=</span> <span class="n">initTable</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="n">Data</span><span class="o">]</span><span class="p">()</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">countTable</span> <span class="o">=</span> <span class="n">initTable</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="kt">int</span><span class="o">]</span><span class="p">()</span>
<span class="k">proc </span><span class="nf">rrdCreate</span><span class="p">(</span><span class="n">file</span><span class="p">,</span> <span class="n">dataSources</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">discard</span> <span class="n">execCmd</span><span class="p">(</span><span class="n">rrdtool</span> <span class="o">&</span> <span class="s">" create "</span> <span class="o">&</span> <span class="n">file</span> <span class="o">&</span> <span class="s">" --step "</span> <span class="o">&</span> <span class="o">$</span><span class="n">freq</span> <span class="o">&</span> <span class="s">" "</span> <span class="o">&</span> <span class="n">dataSources</span> <span class="o">&</span>
<span class="s">" RRA:AVERAGE:0.5:6:480 RRA:AVERAGE:0.5:42:480 RRA:AVERAGE:0.5:147:960"</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">rrdUpdate</span><span class="p">(</span><span class="n">file</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span> <span class="n">values</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="p">`</span><span class="o">$</span><span class="p">`</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">valuesString</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">values</span><span class="p">:</span>
<span class="k">if</span> <span class="n">valuesString</span><span class="p">.</span><span class="n">len</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
<span class="n">valuesString</span><span class="p">.</span><span class="n">add</span> <span class="s">":"</span>
<span class="n">valuesString</span><span class="p">.</span><span class="n">add</span> <span class="n">value</span>
<span class="k">discard</span> <span class="n">execCmd</span><span class="p">(</span><span class="n">rrdtool</span> <span class="o">&</span> <span class="s">" update "</span> <span class="o">&</span> <span class="n">file</span> <span class="o">&</span> <span class="s">" N:"</span> <span class="o">&</span> <span class="n">valuesString</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">updateServer</span><span class="p">(</span><span class="n">server</span><span class="p">:</span> <span class="n">JsonNode</span><span class="p">)</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">domain</span> <span class="o">=</span> <span class="n">server</span><span class="o">[</span><span class="s">"type"</span><span class="o">]</span><span class="p">.</span><span class="n">str</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">dataTable</span><span class="p">.</span><span class="n">mgetOrPut</span><span class="p">(</span><span class="n">domain</span><span class="p">,</span> <span class="n">Data</span><span class="p">()).</span><span class="n">fieldPairs</span><span class="p">:</span>
<span class="k">when</span> <span class="n">value</span> <span class="ow">is</span> <span class="n">BiggestInt</span><span class="p">:</span>
<span class="n">value</span> <span class="o">+=</span> <span class="n">server</span><span class="o">[</span><span class="n">name</span><span class="o">]</span><span class="p">.</span><span class="n">num</span>
<span class="k">elif</span> <span class="n">value</span> <span class="ow">is</span> <span class="kt">float</span><span class="p">:</span>
<span class="n">value</span> <span class="o">+=</span> <span class="n">server</span><span class="o">[</span><span class="n">name</span><span class="o">]</span><span class="p">.</span><span class="n">fnum</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">error</span> <span class="s">"Unhandled type in Data object"</span>
<span class="n">inc</span> <span class="n">countTable</span><span class="p">.</span><span class="n">mgetOrPut</span><span class="p">(</span><span class="n">domain</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="c"># Only save data if we got 30 values in the expected time span</span>
<span class="k">if</span> <span class="n">count</span> <span class="o">==</span> <span class="n">freq</span> <span class="ow">and</span> <span class="n">countTable</span><span class="o">[</span><span class="n">domain</span><span class="o">]</span> <span class="o">==</span> <span class="n">freq</span><span class="p">:</span>
<span class="k">let</span> <span class="n">data</span> <span class="o">=</span> <span class="n">dataTable</span><span class="o">[</span><span class="n">domain</span><span class="o">]</span>
<span class="k">if</span> <span class="n">data</span> <span class="o">==</span> <span class="n">Data</span><span class="p">():</span>
<span class="n">dataTable</span><span class="p">.</span><span class="n">del</span><span class="p">(</span><span class="n">domain</span><span class="p">)</span>
<span class="k">return</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">dataTable</span><span class="o">[</span><span class="n">domain</span><span class="o">]</span> <span class="o">=</span> <span class="n">Data</span><span class="p">()</span>
<span class="k">let</span>
<span class="n">fileNet</span> <span class="o">=</span> <span class="p">(</span><span class="n">rrdDir</span> <span class="o">/</span> <span class="n">domain</span><span class="p">)</span> <span class="o">&</span> <span class="s">"-net.rrd"</span>
<span class="n">fileCpu</span> <span class="o">=</span> <span class="p">(</span><span class="n">rrdDir</span> <span class="o">/</span> <span class="n">domain</span><span class="p">)</span> <span class="o">&</span> <span class="s">"-cpu.rrd"</span>
<span class="n">fileMem</span> <span class="o">=</span> <span class="p">(</span><span class="n">rrdDir</span> <span class="o">/</span> <span class="n">domain</span><span class="p">)</span> <span class="o">&</span> <span class="s">"-mem.rrd"</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">existsFile</span> <span class="n">fileNet</span><span class="p">:</span>
<span class="n">fileNet</span><span class="p">.</span><span class="n">rrdCreate</span><span class="p">(</span><span class="s">"DS:network_rx:GAUGE:60:0:U DS:network_tx:GAUGE:60:0:U"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">existsFile</span> <span class="n">fileCpu</span><span class="p">:</span>
<span class="n">fileCpu</span><span class="p">.</span><span class="n">rrdCreate</span><span class="p">(</span><span class="s">"DS:cpu:GAUGE:60:0:100 DS:load:GAUGE:60:0:U"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">existsFile</span> <span class="n">fileMem</span><span class="p">:</span>
<span class="n">filemem</span><span class="p">.</span><span class="n">rrdCreate</span><span class="p">(</span><span class="s">"DS:memory_used:GAUGE:60:0:U DS:memory_total:GAUGE:60:0:U DS:swap_used:GAUGE:60:0:U DS:swap_total:GAUGE:60:0:U"</span><span class="p">)</span>
<span class="n">fileNet</span><span class="p">.</span><span class="n">rrdUpdate</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">network_rx</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">,</span> <span class="n">data</span><span class="p">.</span><span class="n">network_tx</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">)</span>
<span class="n">fileCpu</span><span class="p">.</span><span class="n">rrdUpdate</span><span class="p">(</span><span class="n">min</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">cpu</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">,</span> <span class="mi">100</span><span class="p">),</span> <span class="n">data</span><span class="p">.</span><span class="n">load</span> <span class="o">/</span> <span class="n">freq</span><span class="p">)</span>
<span class="n">fileMem</span><span class="p">.</span><span class="n">rrdUpdate</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">memory_used</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">,</span> <span class="n">data</span><span class="p">.</span><span class="n">memory_total</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">,</span> <span class="n">data</span><span class="p">.</span><span class="n">swap_used</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">,</span> <span class="n">data</span><span class="p">.</span><span class="n">swap_total</span> <span class="ow">div</span> <span class="n">freq</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">updateAllServers</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">statsJson</span> <span class="o">=</span> <span class="n">parseFile</span> <span class="n">statsJsonFile</span>
<span class="k">let</span> <span class="n">newUpdated</span> <span class="o">=</span> <span class="n">parseBiggestInt</span> <span class="n">statsJson</span><span class="o">[</span><span class="s">"updated"</span><span class="o">]</span><span class="p">.</span><span class="n">str</span>
<span class="k">if</span> <span class="n">newUpdated</span> <span class="o"><=</span> <span class="n">lastUpdated</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">inc</span> <span class="n">count</span>
<span class="k">for</span> <span class="n">server</span> <span class="ow">in</span> <span class="n">statsJson</span><span class="o">[</span><span class="s">"servers"</span><span class="o">]</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">updateServer</span><span class="p">(</span><span class="n">server</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">discard</span>
<span class="k">if</span> <span class="n">count</span> <span class="o">==</span> <span class="n">freq</span><span class="p">:</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">val</span> <span class="ow">in</span> <span class="n">countTable</span><span class="p">.</span><span class="n">mvalues</span><span class="p">:</span>
<span class="n">val</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="k">let</span> <span class="n">startTime</span> <span class="o">=</span> <span class="n">epochTime</span><span class="p">()</span>
<span class="n">updateAllServers</span><span class="p">()</span>
<span class="n">sleep</span><span class="p">(</span><span class="kt">int</span><span class="p">(</span><span class="n">epochTime</span><span class="p">()</span> <span class="o">-</span> <span class="n">startTime</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">)</span> <span class="c"># every second</span></code></pre></figure>
<p>The final graphs can be seen on the <a href="https://ddnet.org/stats/server/">DDNet Server Statistics page</a>.</p>
<p>Nim monitor footprint:</p>
<table>
<thead>
<tr>
<th>Part</th>
<th style="text-align: right">CPU</th>
<th style="text-align: right">Memory</th>
</tr>
</thead>
<tbody>
<tr>
<td>Monitor (9 servers)</td>
<td style="text-align: right">0.03 %</td>
<td style="text-align: right">0.89 MB</td>
</tr>
</tbody>
</table>
<h2 id="alerts-through-cron-and-mail">Alerts through Cron and Mail</h2>
<p>Now we certainly have nice graphs, but automated alerts about suspicious events would be even better, for example:</p>
<ul>
<li>Network traffic over 2 MB/s for 4 min</li>
<li>Memory and swap over 90% for 4 min</li>
<li>CPU over 90% for 21 min</li>
<li>Load over 10 for 21 min</li>
<li>Server unreachable for 1 hour</li>
</ul>
<p>To check these conditions a cron job is run regularly and thanks to a <code class="language-plaintext highlighter-rouge">MAILTO</code> entry a mail is sent when an alert has been triggered.</p>
<p>We can get out a single value from the database to standard output using <code class="language-plaintext highlighter-rouge">PRINT</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">rrdtool graph x <span class="nt">-s-4min</span> <span class="se">\</span>
DEF:v<span class="o">=</span>ddnet.tw-net.rrd:network_rx:AVERAGE <span class="se">\</span>
VDEF:vm<span class="o">=</span>v,AVERAGE <span class="se">\</span>
PRINT:vm:%lf</code></pre></figure>
<p>The alert program itself is written in Nim as well and merely gets a few values from the databases and checks if the limits are exceeded:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">common</span><span class="p">,</span> <span class="n">osproc</span><span class="p">,</span> <span class="n">os</span><span class="p">,</span> <span class="n">json</span><span class="p">,</span> <span class="n">strutils</span>
<span class="k">proc </span><span class="nf">get</span><span class="p">(</span><span class="n">file</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">time</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="kt">float</span> <span class="o">=</span>
<span class="k">let</span> <span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">errorCode</span><span class="p">)</span> <span class="o">=</span> <span class="n">execCmdEx</span><span class="p">(</span><span class="n">rrdtool</span> <span class="o">&</span> <span class="s">" graph x -s -"</span> <span class="o">&</span> <span class="n">time</span> <span class="o">&</span> <span class="s">" DEF:v="</span> <span class="o">&</span> <span class="n">file</span> <span class="o">&</span> <span class="s">":"</span> <span class="o">&</span> <span class="n">value</span> <span class="o">&</span> <span class="s">":AVERAGE VDEF:vm=v,AVERAGE PRINT:vm:%lf"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">errorCode</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">newException</span><span class="p">(</span><span class="n">ValueError</span><span class="p">,</span> <span class="s">"Error code "</span> <span class="o">&</span> <span class="o">$</span><span class="n">errorCode</span> <span class="o">&</span> <span class="s">" from rrdtool: "</span> <span class="o">&</span> <span class="n">output</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">output</span><span class="p">.</span><span class="n">splitLines</span><span class="o">[</span><span class="p">^</span><span class="mi">2</span><span class="o">]</span><span class="p">.</span><span class="n">parseFloat</span>
<span class="k">if</span> <span class="n">paramCount</span><span class="p">()</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"alert [1d|7d|49d]"</span>
<span class="n">quit</span> <span class="mi">1</span>
<span class="k">let</span> <span class="n">statsJson</span> <span class="o">=</span> <span class="n">parseFile</span> <span class="n">statsJsonFile</span>
<span class="k">for</span> <span class="n">server</span> <span class="ow">in</span> <span class="n">statsJson</span><span class="o">[</span><span class="s">"servers"</span><span class="o">]</span><span class="p">:</span>
<span class="k">let</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">server</span><span class="o">[</span><span class="s">"type"</span><span class="o">]</span><span class="p">.</span><span class="n">str</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">server</span><span class="o">[</span><span class="s">"name"</span><span class="o">]</span><span class="p">.</span><span class="n">str</span>
<span class="n">fileNet</span> <span class="o">=</span> <span class="p">(</span><span class="n">rrdDir</span> <span class="o">/</span> <span class="n">domain</span><span class="p">)</span> <span class="o">&</span> <span class="s">"-net.rrd"</span>
<span class="n">fileCpu</span> <span class="o">=</span> <span class="p">(</span><span class="n">rrdDir</span> <span class="o">/</span> <span class="n">domain</span><span class="p">)</span> <span class="o">&</span> <span class="s">"-cpu.rrd"</span>
<span class="n">fileMem</span> <span class="o">=</span> <span class="p">(</span><span class="n">rrdDir</span> <span class="o">/</span> <span class="n">domain</span><span class="p">)</span> <span class="o">&</span> <span class="s">"-mem.rrd"</span>
<span class="k">template</span> <span class="n">alert</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span> <span class="n">echo</span> <span class="n">name</span><span class="p">,</span> <span class="s">" "</span><span class="p">,</span> <span class="n">s</span>
<span class="k">case</span> <span class="n">paramStr</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">of</span> <span class="s">"1d"</span><span class="p">:</span>
<span class="k">if</span> <span class="n">fileNet</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"network_rx"</span><span class="p">,</span> <span class="s">"4min"</span><span class="p">)</span> <span class="o">+</span> <span class="n">fileNet</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"network_tx"</span><span class="p">,</span> <span class="s">"4min"</span><span class="p">)</span> <span class="o">></span> <span class="mi">2_000_000</span><span class="p">:</span>
<span class="n">alert</span> <span class="s">"network traffic over 2 MB/s for 4 min"</span>
<span class="k">if</span> <span class="n">fileMem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"memory_used"</span><span class="p">,</span> <span class="s">"4min"</span><span class="p">)</span> <span class="o">+</span> <span class="n">fileMem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"swap_used"</span><span class="p">,</span> <span class="s">"4min"</span><span class="p">)</span> <span class="o">></span> <span class="mf">0.9</span> <span class="o">*</span> <span class="p">(</span><span class="n">fileMem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"memory_total"</span><span class="p">,</span> <span class="s">"4min"</span><span class="p">)</span> <span class="o">+</span> <span class="n">fileMem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"swap_total"</span><span class="p">,</span> <span class="s">"4min"</span><span class="p">)):</span>
<span class="n">alert</span> <span class="s">"memory and swap over 90% for 4 min"</span>
<span class="k">of</span> <span class="s">"7d"</span><span class="p">:</span>
<span class="k">if</span> <span class="n">fileCpu</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"cpu"</span><span class="p">,</span> <span class="s">"21min"</span><span class="p">)</span> <span class="o">></span> <span class="mf">90.0</span><span class="p">:</span>
<span class="n">alert</span> <span class="s">"CPU over 90% for 21 min"</span>
<span class="k">if</span> <span class="n">fileCpu</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"load"</span><span class="p">,</span> <span class="s">"21min"</span><span class="p">)</span> <span class="o">></span> <span class="mf">10.0</span><span class="p">:</span>
<span class="n">alert</span> <span class="s">"Load over 10 for 21 min"</span>
<span class="k">of</span> <span class="s">"49d"</span><span class="p">:</span>
<span class="k">let</span> <span class="n">network_rx</span> <span class="o">=</span> <span class="n">fileNet</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"network_rx"</span><span class="p">,</span> <span class="s">"4410"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">network_rx</span> <span class="o">!=</span> <span class="n">network_rx</span><span class="p">:</span> <span class="c"># NaN</span>
<span class="n">alert</span> <span class="s">"unreachable for 1 hour"</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"unknown parameter "</span><span class="p">,</span> <span class="n">paramStr</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">quit</span> <span class="mi">1</span></code></pre></figure>
<p>Nim alert footprint:</p>
<table>
<thead>
<tr>
<th>Part</th>
<th style="text-align: right">Runtime</th>
<th style="text-align: right">Memory</th>
</tr>
</thead>
<tbody>
<tr>
<td>Alert (9 servers)</td>
<td style="text-align: right">0.25 s</td>
<td style="text-align: right">2.12 MB</td>
</tr>
</tbody>
</table>
<h2 id="conclusion">Conclusion</h2>
<p>Live view of the last day of the DDNet.tw web server, also hosting this blog:
<img src="https://ddnet.org/stats/server/ddnet.tw-net-1d.png" alt="DDNet.tw Net" />
<img src="https://ddnet.org/stats/server/ddnet.tw-cpu-1d.png" alt="DDNet.tw CPU" />
<img src="https://ddnet.org/stats/server/ddnet.tw-mem-1d.png" alt="DDNet.tw Mem" /></p>
<p>You can see the full graphs on the <a href="https://ddnet.org/stats/server/">DDNet Server Statistics page</a>. As usual you can find the entire source code in our <a href="https://github.com/ddnet/ddnet-scripts/tree/master/rrd">git repository</a>.</p>
<p>All in all the system runs on very little resources, puts out some nice graphs and alerts me automatically about defined problems.</p>
<table>
<thead>
<tr>
<th>Part</th>
<th style="text-align: right">Runtime</th>
<th style="text-align: right">CPU</th>
<th style="text-align: right">Memory</th>
</tr>
</thead>
<tbody>
<tr>
<td>Client</td>
<td style="text-align: right"> </td>
<td style="text-align: right">0.14 %</td>
<td style="text-align: right">3.70 MB</td>
</tr>
<tr>
<td>Server (9 clients)</td>
<td style="text-align: right"> </td>
<td style="text-align: right">0.58 %</td>
<td style="text-align: right">0.18 MB</td>
</tr>
<tr>
<td>Create (9 servers)</td>
<td style="text-align: right">0.01 s</td>
<td style="text-align: right"> </td>
<td style="text-align: right">1.74 MB</td>
</tr>
<tr>
<td>Graph (9 servers)</td>
<td style="text-align: right">1.99 s</td>
<td style="text-align: right"> </td>
<td style="text-align: right">2.82 MB</td>
</tr>
<tr>
<td>Monitor (9 servers)</td>
<td style="text-align: right"> </td>
<td style="text-align: right">0.03 %</td>
<td style="text-align: right">0.89 MB</td>
</tr>
<tr>
<td>Alert (9 servers)</td>
<td style="text-align: right">0.25 s</td>
<td style="text-align: right"> </td>
<td style="text-align: right">2.12 MB</td>
</tr>
</tbody>
</table>
Writing an Async Logger in Nim2016-01-28T00:00:00+01:00https://hookrace.net/blog/writing-an-async-logger-in-nim
<p>Surprisingly I’m working on <a href="/blog/what-is-hookrace/">HookRace</a> again. I might
share a few interesting code snippets and thoughts in this blog along the way.
I’m still going with <a href="http://nim-lang.org/">Nim</a> as the programming language.</p>
<p>For an easy start let’s write a logging module that can be used everywhere in
the game’s server as well as client. There are mostly three aspects that I care
about:</p>
<ul>
<li>Can be configured to write to different files and/or stdout</li>
<li>Writes to the disk asynchronously, preventing any blocking when the disk is
overloaded</li>
<li>Reasonable performance</li>
</ul>
<p>Follow along if you want to witness how fun it is to write code in Nim!</p>
<!--more-->
<h2 id="motivation">Motivation</h2>
<p>A few logger implementations for Nim already exist:</p>
<ul>
<li><a href="http://nim-lang.org/docs/logging.html">logging</a>: A simple logger in the
standard library. Configurable, but no async writing</li>
<li><a href="https://github.com/nim-appkit/omnilog">omnilog</a>: An advanced logger with
lots of features, no async writing</li>
<li><a href="https://github.com/FedericoCeratto/nim-syslog">syslog</a>: A bit too high level
for my taste and no Windows support</li>
</ul>
<p>We could adapt the logging or omnilog modules to our requirements, mainly by
adding async writing. For now let’s write our own simple logging module from
scratch instead. If it turns out to be usable after a few iterations one might
merge it with an existing module. Also, this post would be rather boring if I
just ended up using an existing logger.</p>
<h2 id="implementation">Implementation</h2>
<p>Let’s start with the simplest logger possible:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">log</span><span class="o">*</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="n">echo</span> <span class="n">text</span>
<span class="n">log</span> <span class="s">"Hello World!"</span></code></pre></figure>
<p>We have a procedure <code class="language-plaintext highlighter-rouge">log</code> that is exported (<code class="language-plaintext highlighter-rouge">*</code>). It takes a value <code class="language-plaintext highlighter-rouge">text</code> of
type <code class="language-plaintext highlighter-rouge">string</code> and prints this text to the standard output. Super simple and
can be used like this from another file (or the same one):</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">logger</span>
<span class="n">log</span> <span class="s">"Hello World!"</span></code></pre></figure>
<p>This simply prints <code class="language-plaintext highlighter-rouge">Hello World</code> to the terminal for now when we execute it. We
want to keep using the logger in this exact format.</p>
<p>Next step: Add a fancy logging format:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">times</span>
<span class="k">proc </span><span class="nf">log</span><span class="o">*</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="n">echo</span> <span class="s">"[</span><span class="si">$#</span><span class="s"> </span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">getDateStr</span><span class="p">(),</span> <span class="n">getClockStr</span><span class="p">(),</span> <span class="n">text</span><span class="o">]</span></code></pre></figure>
<p>This uses the <a href="http://nim-lang.org/docs/strutils.html">strutils</a> and
<a href="http://nim-lang.org/docs/times.html">times</a> modules from Nim’s standard
library. The
<a href="http://nim-lang.org/docs/strutils.html#%,string,openArray[string]">%</a>
operator formats the string <code class="language-plaintext highlighter-rouge">"[$# $#]: $#"</code> with a date, clock and our text
into <code class="language-plaintext highlighter-rouge">[2016-01-28 20:04:43]: Hello World!</code></p>
<p>It would also be nice if we could see which module is actually invoking the
logger. The
<a href="http://nim-lang.org/docs/system.html#instantiationInfo,">instantiationInfo</a>
proc can be used to retrieve the filename and line of a template instatiation
at compile time. So let’s make our logger a template, this also improves
performance because we don’t even need a function call at runtime anymore:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">log</span><span class="o">*</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span>
<span class="n">echo</span> <span class="s">"[</span><span class="si">$#</span><span class="s"> </span><span class="si">$#</span><span class="s">][</span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">getDateStr</span><span class="p">(),</span> <span class="n">getClockStr</span><span class="p">(),</span>
<span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="o">]</span></code></pre></figure>
<p>We end up with this output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2016-01-28 20:14:04][example.nim]: Hello World!
</code></pre></div></div>
<p>Let’s get rid of the <code class="language-plaintext highlighter-rouge">.nim</code> at the end:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"> <span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">[0 .. ^5]</code> gives us the part of the string from position 0 (from the start) to
position 5 from the end. New output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2016-01-28 20:14:04][example]: Hello World!
</code></pre></div></div>
<p>Perfect! We’ll keep this exact output format.</p>
<p>Now let’s add the ability to log to multiple files simultaneously. We also just
handle <code class="language-plaintext highlighter-rouge">stdout</code> as a regular file:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">type</span> <span class="n">Logger</span> <span class="o">=</span> <span class="k">object</span>
<span class="n">file</span><span class="p">:</span> <span class="n">File</span>
<span class="k">var</span> <span class="n">loggers</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="n">Logger</span><span class="o">]</span><span class="p">()</span>
<span class="k">proc </span><span class="nf">addLogger</span><span class="o">*</span><span class="p">(</span><span class="n">file</span><span class="p">:</span> <span class="n">File</span><span class="p">)</span> <span class="o">=</span>
<span class="n">loggers</span><span class="p">.</span><span class="n">add</span> <span class="n">Logger</span><span class="p">(</span><span class="n">file</span><span class="p">:</span> <span class="n">file</span><span class="p">)</span>
<span class="k">template</span> <span class="n">log</span><span class="o">*</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">let</span>
<span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span>
<span class="n">str</span> <span class="o">=</span> <span class="s">"[</span><span class="si">$#</span><span class="s"> </span><span class="si">$#</span><span class="s">][</span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">getDateStr</span><span class="p">(),</span> <span class="n">getClockStr</span><span class="p">(),</span>
<span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="o">]</span>
<span class="k">for</span> <span class="n">logger</span> <span class="ow">in</span> <span class="n">loggers</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">writeLine</span> <span class="n">str</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">flushFile</span></code></pre></figure>
<p>By flushing the file we make sure that the output is actually written to the
file right now, not just once the buffer is full. That’s important to us, for
example if the program crashes and we’d lose the last few lines of the log
otherwise.</p>
<p>Now we have to register a few loggers before we can log something, but that’s
the only change:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">logger</span>
<span class="n">addLogger</span> <span class="n">stdout</span>
<span class="n">addLogger</span> <span class="n">open</span><span class="p">(</span><span class="s">"example.log"</span><span class="p">,</span> <span class="n">fmWrite</span><span class="p">)</span>
<span class="n">log</span> <span class="s">"Hello World!"</span></code></pre></figure>
<p>Now we write our log message to stdout as well as a file. If we used the
<code class="language-plaintext highlighter-rouge">fmAppend</code> file mode instead of <code class="language-plaintext highlighter-rouge">fmWrite</code> we could append to the log file,
right now we’re overwriting it every time we run our example program.</p>
<p>Here comes our main problem: What happens if we’re logging while the game
server is running regularly, but the write to the disk takes a few milliseconds
or even longer! The game server gets delayed and our performance is terrible!</p>
<h2 id="adding-a-thread">Adding a Thread</h2>
<p>So let’s add a separate thread to write the log messages. In Nim each thread
has its own heap, so we can’t directly pass a string to another thread. Instead
we will communicate with our new thread using a
<a href="http://nim-lang.org/docs/channels.html">channel</a>, similarly to Go channels. We
start by defining a <code class="language-plaintext highlighter-rouge">Message</code> type to send over our channel:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">type</span>
<span class="n">MessageKind</span> <span class="o">=</span> <span class="k">enum</span>
<span class="n">write</span><span class="p">,</span> <span class="n">update</span><span class="p">,</span> <span class="n">stop</span>
<span class="n">Message</span> <span class="o">=</span> <span class="k">object</span>
<span class="k">case</span> <span class="n">kind</span><span class="p">:</span> <span class="n">MessageKind</span>
<span class="k">of</span> <span class="n">write</span><span class="p">:</span>
<span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="kt">string</span>
<span class="k">of</span> <span class="n">update</span><span class="p">:</span>
<span class="n">loggers</span><span class="p">:</span> <span class="kt">seq</span><span class="o">[</span><span class="n">Logger</span><span class="o">]</span>
<span class="k">of</span> <span class="n">stop</span><span class="p">:</span>
<span class="k">nil</span>
<span class="k">var</span> <span class="n">channel</span><span class="p">:</span> <span class="n">Channel</span><span class="o">[</span><span class="n">Message</span><span class="o">]</span></code></pre></figure>
<p>We define three kinds of messages:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">write</code>: Write a new log message</li>
<li><code class="language-plaintext highlighter-rouge">update</code>: Update the set of files to log to</li>
<li><code class="language-plaintext highlighter-rouge">stop</code>: Stop the logging thread</li>
</ul>
<p>Our new proc for logging looks like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">threadLog</span> <span class="p">{.</span><span class="n">thread</span><span class="p">.}</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">loggers</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="n">Logger</span><span class="o">]</span><span class="p">()</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="k">let</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">recv</span> <span class="n">channel</span>
<span class="k">case</span> <span class="n">msg</span><span class="p">.</span><span class="n">kind</span>
<span class="k">of</span> <span class="n">write</span><span class="p">:</span>
<span class="k">let</span> <span class="n">str</span> <span class="o">=</span> <span class="s">"[</span><span class="si">$#</span><span class="s"> </span><span class="si">$#</span><span class="s">][</span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">getDateStr</span><span class="p">(),</span> <span class="n">getClockStr</span><span class="p">(),</span>
<span class="n">msg</span><span class="p">.</span><span class="n">module</span><span class="p">,</span> <span class="n">msg</span><span class="p">.</span><span class="n">text</span><span class="o">]</span>
<span class="k">for</span> <span class="n">logger</span> <span class="ow">in</span> <span class="n">loggers</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">writeLine</span> <span class="n">str</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">flushFile</span>
<span class="k">of</span> <span class="n">update</span><span class="p">:</span>
<span class="n">loggers</span> <span class="o">=</span> <span class="n">msg</span><span class="p">.</span><span class="n">loggers</span>
<span class="k">of</span> <span class="n">stop</span><span class="p">:</span>
<span class="k">break</span></code></pre></figure>
<p>The thread keeps running until it receives a <code class="language-plaintext highlighter-rouge">stop</code> message. Calling <code class="language-plaintext highlighter-rouge">recv</code> on
a channel blocks until a message is received. Now that the main load is moved
to a new thread, what do we do in our <code class="language-plaintext highlighter-rouge">log</code> template and <code class="language-plaintext highlighter-rouge">addLogger</code> proc?</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">log</span><span class="o">*</span><span class="p">(</span><span class="n">t</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">write</span><span class="p">,</span> <span class="n">module</span><span class="p">:</span> <span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="n">t</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">addLogger</span><span class="o">*</span><span class="p">(</span><span class="n">file</span><span class="p">:</span> <span class="n">File</span><span class="p">)</span> <span class="o">=</span>
<span class="n">loggers</span><span class="p">.</span><span class="n">add</span> <span class="n">Logger</span><span class="p">(</span><span class="n">file</span><span class="p">:</span> <span class="n">file</span><span class="p">)</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">update</span><span class="p">,</span> <span class="n">loggers</span><span class="p">:</span> <span class="n">loggers</span><span class="p">)</span></code></pre></figure>
<p>Simple enough! We also need to initialize the channel and run the thread:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="n">open</span> <span class="n">channel</span>
<span class="k">var</span> <span class="n">thread</span><span class="p">:</span> <span class="n">Thread</span><span class="o">[</span><span class="n">void</span><span class="o">]</span>
<span class="n">thread</span><span class="p">.</span><span class="n">createThread</span> <span class="n">threadLog</span></code></pre></figure>
<p>Now we have to compile with <code class="language-plaintext highlighter-rouge">nim --threads:on c example</code> to get thread support,
or we simply add a <code class="language-plaintext highlighter-rouge">nim.cfg</code> that contains <code class="language-plaintext highlighter-rouge">threads: on</code>. Finally we run our code again and … nothing happens, no log message is printed.</p>
<p>The problem is that we finish the program too quickly. This kills the thread
before it can receive our logging message. So let’s add a proc that manages the
shutdown process more gracefully:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">stopLog</span> <span class="p">{.</span><span class="n">noconv</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">stop</span><span class="p">)</span>
<span class="n">joinThread</span> <span class="n">thread</span>
<span class="n">close</span> <span class="n">channel</span>
<span class="k">for</span> <span class="n">logger</span> <span class="ow">in</span> <span class="n">loggers</span><span class="p">:</span>
<span class="k">if</span> <span class="n">logger</span><span class="p">.</span><span class="n">file</span> <span class="ow">notin</span> <span class="o">[</span><span class="n">stdout</span><span class="p">,</span> <span class="n">stderr</span><span class="o">]</span><span class="p">:</span>
<span class="n">close</span> <span class="n">logger</span><span class="p">.</span><span class="n">file</span>
<span class="n">addQuitProc</span> <span class="n">stopLog</span></code></pre></figure>
<p>That’s it, our logger works! For more convenience we can add log levels and
corresponding templates:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">type</span> <span class="n">Level</span><span class="o">*</span> <span class="p">{.</span><span class="n">pure</span><span class="p">.}</span> <span class="o">=</span> <span class="k">enum</span>
<span class="n">debug</span><span class="p">,</span> <span class="n">info</span><span class="p">,</span> <span class="n">warn</span><span class="p">,</span> <span class="n">error</span><span class="p">,</span> <span class="n">fatal</span>
<span class="k">var</span> <span class="n">logLevel</span><span class="o">*</span> <span class="o">=</span> <span class="n">Level</span><span class="p">.</span><span class="n">info</span>
<span class="k">template</span> <span class="n">debug</span><span class="o">*</span><span class="p">(</span><span class="n">t</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">logLevel</span> <span class="o"><=</span> <span class="n">Level</span><span class="p">.</span><span class="n">debug</span><span class="p">:</span>
<span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">write</span><span class="p">,</span> <span class="n">module</span><span class="p">:</span> <span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="n">t</span><span class="p">)</span>
<span class="k">template</span> <span class="n">info</span><span class="o">*</span><span class="p">(</span><span class="n">t</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">logLevel</span> <span class="o"><=</span> <span class="n">Level</span><span class="p">.</span><span class="n">info</span><span class="p">:</span>
<span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">write</span><span class="p">,</span> <span class="n">module</span><span class="p">:</span> <span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="n">t</span><span class="p">)</span>
<span class="p">...</span></code></pre></figure>
<p>What if we want to log not just strings? We can instead accept any kind and
number of arguments and stringify them automatically using <code class="language-plaintext highlighter-rouge">$</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">send</span><span class="p">(</span><span class="n">module</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span> <span class="n">args</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">write</span><span class="p">,</span> <span class="n">module</span><span class="p">:</span> <span class="n">module</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="s">""</span><span class="p">)</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="n">msg</span><span class="p">.</span><span class="n">text</span><span class="p">.</span><span class="n">add</span> <span class="n">arg</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">msg</span>
<span class="k">template</span> <span class="n">log</span><span class="o">*</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="p">`</span><span class="o">$</span><span class="p">`</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="k">const</span> <span class="n">module</span> <span class="o">=</span> <span class="n">instantiationInfo</span><span class="p">().</span><span class="n">filename</span><span class="o">[</span><span class="mi">0</span> <span class="p">..</span> <span class="p">^</span><span class="mi">5</span><span class="o">]</span>
<span class="n">send</span> <span class="n">module</span><span class="p">,</span> <span class="n">args</span></code></pre></figure>
<p>Now we can call our logger like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">12</span>
<span class="n">info</span> <span class="s">"All "</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="s">" Bananas"</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2016-01-28 21:57:34][example]: All 12 Bananas
</code></pre></div></div>
<p>For more advanced formatting one might want to use the excellent
<a href="http://lyro.bitbucket.org/strfmt/">strfmt</a> library:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">strfmt</span>
<span class="k">var</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">12</span>
<span class="k">var</span> <span class="n">y</span> <span class="o">=</span> <span class="mf">3.4</span>
<span class="n">info</span> <span class="s">"{} + {:.2f} == {}"</span><span class="p">.</span><span class="n">fmt</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">x</span><span class="p">.</span><span class="kt">float</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span>
<span class="n">info</span> <span class="s">interp"I have </span><span class="si">$x</span><span class="s"> Bananas!"</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2016-01-28 22:24:01][example]: 12 + 3.40 == 15.4
[2016-01-28 22:24:01][example]: I have 12 Bananas!
</code></pre></div></div>
<p>Everything works great and we can be sure that hard disk latency will not bite
us. But how is the performance?</p>
<h2 id="optimization">Optimization</h2>
<p>First let’s come up with a simple program that allows us to measure
performance:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">logger</span>
<span class="n">addLogger</span> <span class="n">open</span><span class="p">(</span><span class="s">"bench.log"</span><span class="p">,</span> <span class="n">fmWrite</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">1</span> <span class="p">..</span> <span class="mi">1_000_000</span><span class="p">:</span>
<span class="n">log</span> <span class="s">"Hello World!"</span></code></pre></figure>
<table>
<thead>
<tr>
<th>Build Command</th>
<th style="text-align: right">Runtime</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim c bench</code> (unoptimized)</td>
<td style="text-align: right">31.9 s</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim -d:release c bench</code> (optimized)</td>
<td style="text-align: right">12.1 s</td>
</tr>
</tbody>
</table>
<p>12 seconds to write a million log messages. I guess that’s good enough for my
use case. We’d rather run out of free disk space than have too little
performance like this.</p>
<p>Anyway, let’s see if there are any low hanging fruits. We could look at the
code and try to isolate some parts to figure out what takes long. Or we compile
the benchmark with debugger support using <code class="language-plaintext highlighter-rouge">nim -d:release --debugger:native c
bench</code> and invoke it with <code class="language-plaintext highlighter-rouge">valgrind --tool=callgrind ./bench</code>. Afterwards
kcachegrind can help us analyze the function calls:</p>
<p><img src="/public/kcachegrind.png" style="width:100%; display:inline;" /></p>
<p>What stands out is that <code class="language-plaintext highlighter-rouge">getDateStr</code> and <code class="language-plaintext highlighter-rouge">getClockStr</code> take 33% and 32%
respectively of the total time! Since the time string only changes once every
second we can cache it:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">threadLog</span> <span class="p">{.</span><span class="n">thread</span><span class="p">.}</span> <span class="o">=</span>
<span class="k">var</span>
<span class="n">loggers</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="n">Logger</span><span class="o">]</span><span class="p">()</span>
<span class="n">lastTime</span><span class="p">:</span> <span class="n">Time</span>
<span class="n">timeStr</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="k">let</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">recv</span> <span class="n">channel</span>
<span class="k">case</span> <span class="n">msg</span><span class="p">.</span><span class="n">kind</span>
<span class="k">of</span> <span class="n">write</span><span class="p">:</span>
<span class="k">let</span> <span class="n">newTime</span> <span class="o">=</span> <span class="n">getTime</span><span class="p">()</span>
<span class="k">if</span> <span class="n">newTime</span> <span class="o">!=</span> <span class="n">lastTime</span><span class="p">:</span>
<span class="n">timeStr</span> <span class="o">=</span> <span class="n">getLocalTime</span><span class="p">(</span><span class="n">newTime</span><span class="p">).</span><span class="n">format</span> <span class="s">"yyyy-MM-dd HH:mm:ss"</span>
<span class="n">lastTime</span> <span class="o">=</span> <span class="n">newTime</span>
<span class="k">let</span> <span class="n">str</span> <span class="o">=</span> <span class="s">"[</span><span class="si">$#</span><span class="s">][</span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">timeStr</span><span class="p">,</span> <span class="n">msg</span><span class="p">.</span><span class="n">module</span><span class="p">,</span> <span class="n">msg</span><span class="p">.</span><span class="n">text</span><span class="o">]</span>
<span class="k">for</span> <span class="n">logger</span> <span class="ow">in</span> <span class="n">loggers</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">writeLine</span> <span class="n">str</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">flushFile</span>
<span class="p">...</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">getTime()</code> gets the time in seconds, so we only format a new <code class="language-plaintext highlighter-rouge">timeStr</code> when a
new second has arrived.</p>
<table>
<thead>
<tr>
<th>Code Change</th>
<th style="text-align: right">Runtime</th>
</tr>
</thead>
<tbody>
<tr>
<td>Cache time string</td>
<td style="text-align: right">4.4 s</td>
</tr>
</tbody>
</table>
<p>Flushing the file after each write still consumes a lot of time. Unfortunately this
can’t be seen using valgrind because valgrind’s instrumentation slows down the
entire program so much that the flushing itself becomes insignificant.</p>
<p>The trick is to flush the file only when we’re actually done writing, that
means when we have no more messages waiting in the channel:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">if</span> <span class="n">channel</span><span class="p">.</span><span class="n">peek</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">flushFile</span></code></pre></figure>
<table>
<thead>
<tr>
<th>Code Change</th>
<th style="text-align: right">Runtime</th>
</tr>
</thead>
<tbody>
<tr>
<td>Flush only when necessary</td>
<td style="text-align: right">2.1 s</td>
</tr>
</tbody>
</table>
<p>A general tip when optimizing Nim code is to prevent string allocations and
copying. Instead of using <code class="language-plaintext highlighter-rouge">writeLine</code> we can add the newline directly and call
<code class="language-plaintext highlighter-rouge">write</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">let</span> <span class="n">str</span> <span class="o">=</span> <span class="s">"[</span><span class="si">$#</span><span class="s">][</span><span class="si">$#</span><span class="s">]: </span><span class="si">$#</span><span class="se">\n</span><span class="s">"</span> <span class="o">%</span> <span class="o">[</span><span class="n">timeStr</span><span class="p">,</span> <span class="n">msg</span><span class="p">.</span><span class="n">module</span><span class="p">,</span> <span class="n">msg</span><span class="p">.</span><span class="n">text</span><span class="o">]</span>
<span class="k">for</span> <span class="n">logger</span> <span class="ow">in</span> <span class="n">loggers</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">write</span> <span class="n">str</span></code></pre></figure>
<p>We can use the same message and simply reset its text string instead of
allocating a new one every time:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span><span class="n">kind</span><span class="p">:</span> <span class="n">write</span><span class="p">,</span> <span class="n">system</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="s">""</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">send</span><span class="p">(</span><span class="n">moduleName</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span> <span class="n">args</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="kt">string</span><span class="p">,</span> <span class="p">`</span><span class="o">$</span><span class="p">`</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="n">msg</span><span class="p">.</span><span class="n">system</span> <span class="o">=</span> <span class="n">moduleName</span>
<span class="n">msg</span><span class="p">.</span><span class="n">text</span><span class="p">.</span><span class="n">setLen</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="n">msg</span><span class="p">.</span><span class="n">text</span><span class="p">.</span><span class="n">add</span> <span class="n">arg</span>
<span class="n">channel</span><span class="p">.</span><span class="n">send</span> <span class="n">msg</span></code></pre></figure>
<p>These two changes don’t have a huge effect, but it’s better than nothing:</p>
<table>
<thead>
<tr>
<th>Code Change</th>
<th style="text-align: right">Runtime</th>
</tr>
</thead>
<tbody>
<tr>
<td>Prevent allocations</td>
<td style="text-align: right">1.8 s</td>
</tr>
</tbody>
</table>
<p>Another idea is to use a non-locking <code class="language-plaintext highlighter-rouge">write</code> since we only write from one
thread anyway. The POSIX C function <code class="language-plaintext highlighter-rouge">fwrite_unlocked</code> can easily be imported
and used in Nim:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">fwriteUnlocked</span><span class="p">(</span><span class="n">buf</span><span class="p">:</span> <span class="n">pointer</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">n</span><span class="p">:</span> <span class="kt">int</span><span class="p">,</span> <span class="n">f</span><span class="p">:</span> <span class="n">File</span><span class="p">):</span> <span class="kt">int</span> <span class="p">{.</span>
<span class="n">importc</span><span class="p">:</span> <span class="s">"fwrite_unlocked"</span><span class="p">,</span> <span class="n">noDecl</span><span class="p">.}</span>
<span class="k">proc </span><span class="nf">writeUnlocked</span><span class="p">(</span><span class="n">f</span><span class="p">:</span> <span class="n">File</span><span class="p">,</span> <span class="n">s</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">fwriteUnlocked</span><span class="p">(</span><span class="n">cstring</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="n">s</span><span class="p">.</span><span class="n">len</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="o">!=</span> <span class="n">s</span><span class="p">.</span><span class="n">len</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">newException</span><span class="p">(</span><span class="n">IOError</span><span class="p">,</span> <span class="s">"cannot write string to file"</span><span class="p">)</span>
<span class="n">logger</span><span class="p">.</span><span class="n">file</span><span class="p">.</span><span class="n">writeUnlocked</span> <span class="n">str</span></code></pre></figure>
<p>Unfortunately this does not have any effect on performance.</p>
<p>So now we can log <code class="language-plaintext highlighter-rouge">"Hello World"</code> a million times in 1.8 seconds. For
comparison, simply printing <code class="language-plaintext highlighter-rouge">"Hello World"</code> a million times, without any
formatting or asynchronous writing, takes 2.4 seconds already when redirecting
stdout to a file:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">1</span> <span class="p">..</span> <span class="mi">1_000_000</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Hello World!"</span></code></pre></figure>
<p>This is of course because of the flushing that happens internally in <code class="language-plaintext highlighter-rouge">echo</code>. If
we opt to use a raw <code class="language-plaintext highlighter-rouge">write</code> directly we can write the million strings in just
0.2 s, so there is still headroom for further improvements.</p>
<h2 id="final-words">Final words</h2>
<p>We implemented a simple logger, saw a few cool Nim features along the way and
finally improved the performance by a factor of 6.</p>
<p>An even better optimization would be not to use channels for inter-thread
communication. After all even the <a href="http://nim-lang.org/docs/channels.html">channels
documentation</a> notes that they are
slow. Instead we could create a data structue in the heap shared between
threads and use this. But I doubt that I will need more performance for
logging, so this is good enough for now.</p>
<p>The resulting code of this post and future HookRace code can be found on
<a href="https://github.com/hookrace/hookrace/">GitHub</a>.</p>
DDNet Live: Twitch spectates an online game2016-01-10T00:00:00+01:00https://hookrace.net/blog/ddnet-live
<p>Last night I had an idea and implemented it, soo let’s see what will happen.
But first, the idea:</p>
<blockquote>
<p>Have a livestream [1] of <a href="https://ddnet.org/">DDNet</a> running non-stop [2] that
always shows some interesting [3] players on the server.</p>
</blockquote>
<p>The resulting livestream is running on <a href="https://www.twitch.tv/ddnetlive">Twitch</a>.
All the scripts are on <a href="https://github.com/ddnet/ddnet-live">Github</a></p>
<!--more-->
<h2 id="1-livestream">1. Livestream</h2>
<p>It’s surprisingly simple to livestream from Linux to Twitch. Only FFmpeg is needed:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/sh</span>
<span class="nv">INRES</span><span class="o">=</span><span class="s2">"1280x720"</span> <span class="c"># input resolution</span>
<span class="nv">FPS</span><span class="o">=</span><span class="s2">"30"</span> <span class="c"># target FPS</span>
<span class="nv">GOP</span><span class="o">=</span><span class="s2">"60"</span> <span class="c"># i-frame interval, should be double of FPS, </span>
<span class="nv">GOPMIN</span><span class="o">=</span><span class="s2">"30"</span> <span class="c"># min i-frame interval, should be equal to fps, </span>
<span class="nv">THREADS</span><span class="o">=</span><span class="s2">"2"</span>
<span class="nv">CBR</span><span class="o">=</span><span class="s2">"2000k"</span> <span class="c"># constant bitrate (should be between 1000k - 3000k)</span>
<span class="nv">QUALITY</span><span class="o">=</span><span class="s2">"ultrafast"</span> <span class="c"># one of the many FFMPEG preset</span>
<span class="nv">AUDIO_SRATE</span><span class="o">=</span><span class="s2">"44100"</span>
<span class="nv">AUDIO_CHANNELS</span><span class="o">=</span><span class="s2">"2"</span> <span class="c"># 1 for mono output, 2 for stereo</span>
<span class="nv">AUDIO_ERATE</span><span class="o">=</span><span class="s2">"96k"</span> <span class="c"># audio encoding rate</span>
<span class="nv">SERVER</span><span class="o">=</span><span class="s2">"live-fra"</span> <span class="c"># http://bashtech.net/twitch/ingest.php for list</span>
<span class="nb">source</span> ./secret.sh
ffmpeg <span class="nt">-v</span> 0 <span class="nt">-f</span> x11grab <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$INRES</span><span class="s2">"</span> <span class="nt">-r</span> <span class="s2">"</span><span class="nv">$FPS</span><span class="s2">"</span> <span class="nt">-i</span> :0.0 <span class="se">\</span>
<span class="nt">-f</span> alsa <span class="nt">-i</span> pulse <span class="nt">-f</span> flv <span class="nt">-ac</span> <span class="nv">$AUDIO_CHANNELS</span> <span class="se">\</span>
<span class="nt">-b</span>:a <span class="nv">$AUDIO_ERATE</span> <span class="nt">-ar</span> <span class="nv">$AUDIO_SRATE</span> <span class="se">\</span>
<span class="nt">-vcodec</span> libx264 <span class="nt">-g</span> <span class="nv">$GOP</span> <span class="nt">-keyint_min</span> <span class="nv">$GOPMIN</span> <span class="nt">-b</span>:v <span class="nv">$CBR</span> <span class="se">\</span>
<span class="nt">-minrate</span> <span class="nv">$CBR</span> <span class="nt">-maxrate</span> <span class="nv">$CBR</span> <span class="nt">-vf</span> <span class="s2">"format=yuv420p"</span><span class="se">\</span>
<span class="nt">-preset</span> <span class="nv">$QUALITY</span> <span class="nt">-acodec</span> libmp3lame <span class="nt">-threads</span> <span class="nv">$THREADS</span> <span class="se">\</span>
<span class="nt">-strict</span> normal <span class="nt">-bufsize</span> <span class="nv">$CBR</span> <span class="se">\</span>
<span class="s2">"rtmp://</span><span class="nv">$SERVER</span><span class="s2">.twitch.tv/app/</span><span class="nv">$STREAM_KEY</span><span class="s2">"</span></code></pre></figure>
<p>This works pretty well, but takes quite a bit of CPU on my old computer. So
instead I wanted to run it on my new server with a Haswell-era J1900 CPU.</p>
<p>Possible ways to improve performance:</p>
<ul>
<li>Use an OpenGL recorder instead of x11grab</li>
<li>Get Intel Quick Sync Video working for hardware H264 encoding</li>
</ul>
<h2 id="2-running-non-stop">2. Running Non-Stop</h2>
<p>My server is a small and cheap ASRock Q1900-ITX:</p>
<p><img src="/public/q1900-itx.jpg" alt="Q1900-ITX" /></p>
<p>The nice part is that it’s passively cooled and quite power efficient, drawing
only 5 Watts in idle. This machine has been running as my home server for quite
some time, but barely gets any action. Let’s change that!</p>
<p>The X server starts without problems even without any monitors attached, the
only thing that’s left to do is increasing the framebuffer size so that our
game can run in it:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">cat </span>ddnet.sh
<span class="c">#!/bin/sh</span>
pulseaudio <span class="nt">--start</span>
xrandr <span class="nt">--fb</span> 1280x720 <span class="c"># Adjust framebuffer</span>
<span class="nb">cp </span>settings_ddnet.cfg ~/.teeworlds/ <span class="c"># Restore backup</span>
<span class="nb">cd </span>ddnet <span class="o">&&</span> ./DDNet
<span class="nv">$ </span>xinit ddnet.sh</code></pre></figure>
<p>We don’t even need a window manager. After all we just run a single window in
exactly the resolution of the framebuffer.</p>
<p>The FFmpeg recording still works in exactly the same way. Unfortunately I
couldn’t get H264 hardware encoding to work with the J1900 CPU. <a href="https://github.com/shenhailuanma/qsv-ffmpeg-codec/issues/3">Related bug
reports</a> make me
believe it just doesn’t work on these cheaper Intel CPUs.</p>
<h2 id="3-artificial-intelligence-twitch-control">3. <s>Artificial Intelligence</s> Twitch Control</h2>
<p>Now that we have the game running and are streaming it to Twitch, we need to
control it somehow. My goal was to find an approach that always shows some
interesting players in action, so that you could watch the stream all day and
enjoy it.</p>
<p>But finding a reasonable way to do this seems too complicated and I didn’t look
forward to hacking the DDNet C/C++ source code, so instead I opted to utilize
an existing system in DDNet: FIFO command input!</p>
<p>DDNet servers and clients can be remote controlled through a FIFO file. This is
very useful to send the same commands to dozens of servers at once. But for the
client its use was pretty limited, until now!</p>
<p>Instead of thinking of an algorithm to find interesting players, why not let
the Twitch viewers themselves control who they want to watch through the chat?</p>
<p>I did not modify the DDNet source code in any way and instead wrote a small
<a href="http://nim-lang.org/">Nim</a> script based on the <a href="https://github.com/nim-lang/irc">IRC
module</a> to connect to Twitch’s IRC server and
forward the commands to the FIFO:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">irc</span><span class="p">,</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">secret</span>
<span class="k">const</span> <span class="n">forbiddenCommands</span> <span class="o">=</span> <span class="o">@[</span><span class="s">"exec"</span><span class="p">,</span> <span class="s">"quit"</span><span class="p">,</span> <span class="s">"exit"</span><span class="p">,</span> <span class="s">"disconnect"</span><span class="o">]</span>
<span class="k">var</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">newIrc</span><span class="p">(</span><span class="s">"irc.twitch.tv"</span><span class="p">,</span> <span class="n">nick</span> <span class="o">=</span> <span class="s">"ddnetlive"</span><span class="p">,</span>
<span class="n">serverPass</span> <span class="o">=</span> <span class="n">serverPass</span><span class="p">,</span> <span class="n">joinChans</span> <span class="o">=</span> <span class="o">@[</span><span class="s">"#ddnetlive"</span><span class="o">]</span><span class="p">)</span>
<span class="n">fifo</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="s">"input.fifo"</span><span class="p">,</span> <span class="n">fmWrite</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">handle</span><span class="p">(</span><span class="n">nick</span><span class="p">,</span> <span class="n">cmd</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">forbiddenCommands</span><span class="p">:</span>
<span class="k">if</span> <span class="n">cmd</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">return</span>
<span class="n">echo</span> <span class="n">nick</span><span class="p">,</span> <span class="s">": "</span><span class="p">,</span> <span class="n">cmd</span>
<span class="n">stdout</span><span class="p">.</span><span class="n">flushFile</span><span class="p">()</span>
<span class="n">fifo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
<span class="n">fifo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="n">fifo</span><span class="p">.</span><span class="n">flushFile</span><span class="p">()</span>
<span class="n">client</span><span class="p">.</span><span class="n">connect</span><span class="p">()</span>
<span class="k">var</span> <span class="n">event</span><span class="p">:</span> <span class="n">TIrcEvent</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="k">if</span> <span class="n">client</span><span class="p">.</span><span class="n">poll</span><span class="p">(</span><span class="n">event</span><span class="p">):</span>
<span class="k">case</span> <span class="n">event</span><span class="p">.</span><span class="n">typ</span>
<span class="k">of</span> <span class="n">EvConnected</span><span class="p">:</span>
<span class="k">discard</span>
<span class="k">of</span> <span class="n">EvDisconnected</span><span class="p">,</span> <span class="n">EvTimeout</span><span class="p">:</span>
<span class="n">client</span><span class="p">.</span><span class="n">reconnect</span><span class="p">()</span>
<span class="k">of</span> <span class="n">EvMsg</span><span class="p">:</span>
<span class="k">case</span> <span class="n">event</span><span class="p">.</span><span class="n">cmd</span>
<span class="k">of</span> <span class="n">MPrivMsg</span><span class="p">:</span>
<span class="n">handle</span><span class="p">(</span><span class="n">event</span><span class="p">.</span><span class="n">nick</span><span class="p">,</span> <span class="n">event</span><span class="p">.</span><span class="n">params</span><span class="o">[</span><span class="p">^</span><span class="mi">1</span><span class="o">]</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">discard</span></code></pre></figure>
<p>Right now there is really not much limitation to what you can do. Only a
handful of commands are explicitly blocked, otherwise the client can be
controlled freely and every single chat message is sent to the client. The
<a href="https://ddnet.org/settingscommands/#client-commands">client commands</a> and
<a href="https://ddnet.org/settingscommands/#client-settings">client settings</a> list the
available commands and settings. <a href="https://ddnet.org/settingscommands/#chat-commands">Chat
commands</a> can be sent to the
server as well through the <code class="language-plaintext highlighter-rouge">say</code> command. Some examples:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">connect ger.ddnet.org:8303
team <span class="nt">-1</span> <span class="c"># Join spectators</span>
spectate_next <span class="c"># Spectate the next player</span>
spectate 0 <span class="c"># Spectate player with ID 0</span>
say Hi from twitch.tv/ddnetlive <span class="c"># Write chat messages</span>
player_name DDNetLive <span class="c"># Change name</span></code></pre></figure>
<p>Let’s see how this goes. Luckily the Nim script runs independently from the
server, so I will be able to make changes to it on the fly. Just be nice and
don’t cause any trouble, thanks.</p>
<p>Feel free to <a href="https://twitch.tv/ddnetlive">head over to Twitch</a>, watch the
action and control the server using the Twitch chat. Twitch has some delay, so
it takes a few seconds for you to see your command executed.</p>
Nim binary size from 160 KB to 150 Bytes2015-05-04T00:00:00+02:00https://hookrace.net/blog/nim-binary-size
<p>The size of binaries in the <a href="http://nim-lang.org/">Nim</a> programming language seems <a href="https://www.schipplock.software/p/static-linking-with-nim.html">to be a</a> <a href="http://forum.nim-lang.org/t/679">popular</a> <a href="http://forum.nim-lang.org/t/963">topic</a> <a href="http://forum.nim-lang.org/t/1171">recently</a>. Nim’s slogan is <em>expressive, efficient, elegant</em>, so let’s examine the <em>efficient</em> part in this post by exploring a few ways to reduce the size of a simple Nim <code class="language-plaintext highlighter-rouge">Hello World</code> binary on Linux. Along the way we will:</p>
<ul>
<li>Build a regular program into a 6 KB binary</li>
<li>Disregard the C standard library and build a 952 byte binary</li>
<li>Use a custom linker script and ELF header to build a 150 byte binary <br />(<a href="http://mainisusuallyafunction.blogspot.de/2015/01/151-byte-static-linux-binary-in-rust.html">1 byte smaller than in Rust</a>)</li>
</ul>
<p>The full source code of this post can be found in <a href="https://github.com/def-/nim-binary-size">its repository</a>. All measurements are done on Linux x86-64 with GCC 5.1 and Clang 3.6.0.</p>
<!--more-->
<h2 id="using-the-c-standard-library">Using the C Standard Library</h2>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="n">echo</span> <span class="s">"Hello!"</span></code></pre></figure>
<p>By default Nim uses GCC as the backend C compiler on most platforms, and we dynamically link against glibc. We can try optimizing for speed and size, as well as strip unnecessary symbols after the compilation:</p>
<table>
<thead>
<tr>
<th>Command (GCC backend)</th>
<th style="text-align: right">Binary Size</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim c hello</code></td>
<td style="text-align: right">160 KB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim -d:release c hello</code></td>
<td style="text-align: right">61 KB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim -d:release --opt:size c hello</code></td>
<td style="text-align: right">25 KB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim -d:release --opt:size c hello && strip -s hello</code></td>
<td style="text-align: right">19 KB</td>
</tr>
</tbody>
</table>
<p>That’s pretty nice and can be done with any Nim program to reduce binary size.</p>
<p>Now let’s try to get rid of glibc, at least temporarily (we will come back to a more permanent solution later). Instead of glibc we’re now statically linking against musl libc:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --gcc.exe:/usr/local/musl/bin/musl-gcc \
--gcc.linkerexe:/usr/local/musl/bin/musl-gcc \
-d:release --opt:size --passL:-static c hello
$ strip -s hello
30 KB
</code></pre></div></div>
<p>Update: The order of arguments to nim matters, <code class="language-plaintext highlighter-rouge">--passL:-static</code> has to be passed after setting the gcc exe so that it isn’t overwritten.</p>
<p>So that’s a statically linked binary in 30 KB, which can be deployed without depending on any glibc version (or any other libraries)!</p>
<p>What about using Clang instead of GCC by setting <code class="language-plaintext highlighter-rouge">--cc:clang</code>:</p>
<table>
<thead>
<tr>
<th>Command (Clang backend)</th>
<th style="text-align: right">Binary Size</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim --cc:clang c hello</code></td>
<td style="text-align: right">168 KB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim --cc:clang -d:release c hello</code></td>
<td style="text-align: right">33 KB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim --cc:clang -d:release --opt:size c hello</code></td>
<td style="text-align: right">29 KB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">nim --cc:clang -d:release --opt:size c hello && strip -s hello</code></td>
<td style="text-align: right">23 KB</td>
</tr>
</tbody>
</table>
<p>The speed optimized binary is much smaller, but the one optimized for size isn’t. The exact behaviour of Clang and GCC of course depends on their version, so expect to see (at least) slightly different numbers on your system.</p>
<p>Looks like the GCC backend is a better choice currently, so let’s try stripping down the binary further with it:</p>
<p>As a first step we disable the garbage collector, not like we need it for this program anyway:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --gc:none -d:release --opt:size c hello
$ strip -s hello
11 KB
</code></pre></div></div>
<p>Next we remove all the nice dynamic memory, error handling and other OS dependent goodies with <code class="language-plaintext highlighter-rouge">--os:standalone</code> (this implies <code class="language-plaintext highlighter-rouge">--gc:none</code>). We have to provide a <code class="language-plaintext highlighter-rouge">panicoverride.nim</code> that contains these two procs, about which we don’t care anyway. Who needs error handling when you can have a 6 KB binary instead:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">rawoutput</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span> <span class="k">discard</span>
<span class="k">proc </span><span class="nf">panic</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="kt">string</span><span class="p">)</span> <span class="o">=</span> <span class="k">discard</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --os:standalone -d:release c hello
$ strip -s hello
6.1 KB
</code></pre></div></div>
<h2 id="disregarding-the-c-standard-library">Disregarding the C standard library</h2>
<p>Now we have to start thinking more out of the box: If we want a program that really does nothing, not even print <code class="language-plaintext highlighter-rouge">Hello!</code>, we can just use an empty file. Now we don’t rely on the C standard library anymore and can try to exclude it totally by using <code class="language-plaintext highlighter-rouge">-passL:-nostdlib</code> (passL simply passes that argument to GCC at the linking step):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --os:standalone -d:release --passL:-nostdlib c hello
CC: hello
CC: stdlib_system
[Linking]
ld: warning: cannot find entry symbol _start; defaulting to 0000000000400160
$ strip -s hello
1.4 KB
</code></pre></div></div>
<p>Wow, that’s small! Let’s run our program, that does nothing, and enjoy:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./hello
Segmentation fault (core dumped)
</code></pre></div></div>
<p>Ouch. That didn’t go so well. Take another look at the linker output and notice what the problem is: We can’t just expect our code to be run, the binary starts its execution at some random, wrong, position. Instead we have to take over the work of the C standard library now and supply our own <code class="language-plaintext highlighter-rouge">_start</code> function:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">syscall</span>
<span class="k">proc </span><span class="nf">main</span> <span class="p">{.</span><span class="n">exportc</span><span class="p">:</span> <span class="s">"_start"</span><span class="p">.}</span> <span class="o">=</span>
<span class="k">discard</span> <span class="n">syscall</span><span class="p">(</span><span class="n">EXIT</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span></code></pre></figure>
<p>We also have to explicitly exit the program, for which we use my <a href="https://github.com/def-/nim-syscall">syscall library</a>, that provides raw system calls into the Linux kernel in Nim. Let’s wrap the syscalls we need and use them to write proper Nim code with them:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">syscall</span>
<span class="k">const</span> <span class="n">STDOUT</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">proc </span><span class="nf">write</span><span class="p">(</span><span class="n">fd</span><span class="p">:</span> <span class="n">cint</span><span class="p">,</span> <span class="n">buf</span><span class="p">:</span> <span class="n">cstring</span><span class="p">,</span> <span class="n">len</span><span class="p">:</span> <span class="n">csize</span><span class="p">):</span> <span class="n">clong</span>
<span class="p">{.</span><span class="n">inline</span><span class="p">,</span> <span class="n">discardable</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">syscall</span><span class="p">(</span><span class="n">WRITE</span><span class="p">,</span> <span class="n">fd</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="n">len</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">exit</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="n">clong</span><span class="p">):</span> <span class="n">clong</span> <span class="p">{.</span><span class="n">inline</span><span class="p">,</span> <span class="n">discardable</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">syscall</span><span class="p">(</span><span class="n">EXIT</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">main</span> <span class="p">{.</span><span class="n">exportc</span><span class="p">:</span> <span class="s">"_start"</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">write</span> <span class="n">STDOUT</span><span class="p">,</span> <span class="s">"Hello!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="mi">7</span>
<span class="n">exit</span> <span class="mi">0</span></code></pre></figure>
<p>Now we can successfully compile like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --os:standalone -d:release --passL:-nostdlib --noMain c hello
$ strip -s hello
1.5 KB
$ ./hello
Hello!
</code></pre></div></div>
<p>The final trick of this section is to tell GCC to optimize away unused functions. These functions are the initializations of Nim modules, like our <code class="language-plaintext highlighter-rouge">hello</code> module or the <code class="language-plaintext highlighter-rouge">system</code> module from the standard library, but they just end up empty here anyway. Maybe the Nim compiler could leave them out on its own, but usually you don’t care about saving a few bytes and have something more important to do. Today we care, so first we tell GCC to put function and data items into separate sections (<code class="language-plaintext highlighter-rouge">-ffunction-sections</code> & <code class="language-plaintext highlighter-rouge">-fdata-sections</code>) at the compile step. At the linking step we tell Nim to tell GCC to pass <code class="language-plaintext highlighter-rouge">--gc-sections</code> to <code class="language-plaintext highlighter-rouge">ld</code>, our linker, which then removes sections that are not referenced anywhere:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --os:standalone -d:release --passL:-nostdlib --noMain \
--passC:-ffunction-sections --passC:-fdata-sections \
--passL:-Wl,--gc-sections c hello
$ strip -s hello
952 B
</code></pre></div></div>
<p>Great! We’re down from an initial binary size of 160 KB to just 952 bytes. Can we get even smaller? Yes, indeed, but not with the default tooling.</p>
<h2 id="custom-linking-to-achieve-150-bytes">Custom Linking to achieve 150 Bytes</h2>
<p>This uses the exact method from the <a href="http://mainisusuallyafunction.blogspot.de/2015/01/151-byte-static-linux-binary-in-rust.html">151-byte static Linux binary in Rust</a> blog post, except that Nim with GCC manage to get down 1 byte more. Meanwhile Clang needs 1 byte more than the Rust version.</p>
<p>We continue with the exact same program that we just got down to 952 bytes. But instead of letting Nim do all the work (with quite a lot of options by now) we simply create an object file from Nim (<code class="language-plaintext highlighter-rouge">--app:staticlib</code>), and go manually from there. A <a href="https://github.com/def-/nim-binary-size/blob/master/script.ld">custom linker script</a> and a <a href="https://github.com/def-/nim-binary-size/blob/master/elf.s">custom ELF header</a> do the main work. But the actual code that’s executed is still fully provided by our Nim code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --app:staticlib --os:standalone -d:release --noMain \
--passC:-ffunction-sections --passC:-fdata-sections \
--passL:-Wl,--gc-sections c hello
$ ld --gc-sections -e _start -T script.ld -o payload hello.o
$ objcopy -j combined -O binary payload payload.bin
$ ENTRY=$(nm -f posix payload | grep '^_start ' | awk '{print $3}')
$ nasm -f bin -o hello -D entry=0x$ENTRY elf.s
$ chmod +x hello
$ wc -c < hello
158
$ ./hello
Hello!
</code></pre></div></div>
<p>158 bytes! There’s a final trick to get 8 bytes smaller. We put our string inside the padding in the <a href="https://github.com/def-/nim-binary-size/blob/master/elf.s">ELF header</a> and access that memory manually:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">main</span> <span class="p">{.</span><span class="n">exportc</span><span class="p">:</span> <span class="s">"_start"</span><span class="p">.}</span> <span class="o">=</span>
<span class="n">write</span> <span class="n">STDOUT</span><span class="p">,</span> <span class="k">cast</span><span class="o">[</span><span class="n">cstring</span><span class="o">]</span><span class="p">(</span><span class="mh">0x00400008</span><span class="p">),</span> <span class="mi">7</span>
<span class="n">exit</span> <span class="mi">0</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ wc -c < hello
150
$ ./hello
Hello!
</code></pre></div></div>
<p>150 bytes! That’s the final size we’re going to get. If you still don’t have enough and want to write your binary manually you can use even more hacks to get down to 45 bytes as described in the excellent <a href="http://www.muppetlabs.com/%7Ebreadbox/software/tiny/teensy.html">Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux</a>.</p>
<h2 id="summary">Summary</h2>
<p>Nim is fine for writing small binaries. Now you also know how to write Nim without the C standard library. It might be fun to write your own runtime in Nim from scratch. You can check out the <a href="https://github.com/def-/nim-binary-size/">repository</a> to get your own results:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./run.sh
== Using the C Standard Library ==
hello_unoptimized 163827
hello_release 62131
hello_optsize 25248
hello_optsize_strip 18552
hello_gcnone 10344
hello_standalone 6208
== Disregarding the C Standard Library ==
hello2 1776
hello3 952
== Custom Linking ==
hello3_custom 158
hello4_custom 150
$ objdump -rd nimcache/hello4.o
...
0000000000000000 <_start>:
0: b8 01 00 00 00 mov $0x1,%eax
5: ba 07 00 00 00 mov $0x7,%edx
a: be 08 00 40 00 mov $0x400008,%esi
f: 48 89 c7 mov %rax,%rdi
12: 0f 05 syscall
14: 31 ff xor %edi,%edi
16: b8 3c 00 00 00 mov $0x3c,%eax
1b: 0f 05 syscall
1d: c3 retq
...
</code></pre></div></div>
<p>Discussions on <a href="https://news.ycombinator.com/item?id=9485526">Hacker News</a> and <a href="https://www.reddit.com/r/programming/comments/34tb7r/nim_binary_size_from_160_kb_to_150_bytes/">Reddit</a>.</p>
<h3 id="addendum">Addendum</h3>
<p>I did this for 32bit x86 now as well, which results in a 119 byte binary with GCC and 118 byte with Clang, more information in the <a href="https://github.com/def-/nim-binary-size#x86">repository</a>.</p>
<p>With <a href="https://github.com/Araq/Nim/pull/2657">a simple patch</a> to the Nim compiler the <code class="language-plaintext highlighter-rouge">{.noReturn.}</code> pragma now actually removes the final <code class="language-plaintext highlighter-rouge">retq</code> call that is useless after the <code class="language-plaintext highlighter-rouge">EXIT</code> syscall. So the final binary size now is 149 bytes with x86-64, 116 bytes with x86.</p>
Porting a NES emulator from Go to Nim2015-05-01T00:00:00+02:00https://hookrace.net/blog/porting-nes-go-nim
<blockquote>
<p>Let me get this straight. We have an emulator for 1985 hardware that was written in a pretty new language (Go), ported to a language that isn’t even 1.0 (Nim), compiled to C, then compiled to JavaScript? And the damn thing actually works? That’s kind of amazing.</p>
<p>— <cite>Summary by <a href="https://news.ycombinator.com/item?id=9474030">haberman</a></cite></p>
</blockquote>
<!--more-->
<p>I spent the last weeks working on <a href="https://github.com/def-/nimes">NimES</a>, a NES emulator in the <a href="http://nim-lang.org/">Nim</a> programming language. As I really liked <a href="https://github.com/fogleman/nes">fogleman’s NES emulator in Go</a> I ended up mostly porting it to Nim. The source code is so clean that it’s often easier to understand the internals of the NES by reading the source code than by reading documentation about it.</p>
<p>The choice of backend fell on SDL2 for me, contrary to GLFW + PortAudio that the Go version used. This was mainly motivated by the great portability promised by SDL2. Later we will see how porting to JavaScript and Android worked. If you’re impatient and want to play a game, there’s a <a href="/nimes/">JS demo</a>.</p>
<h2 id="comparison-of-go-and-nim">Comparison of Go and Nim</h2>
<p>Most Go concepts are quite trivial to translate to Nim. This made the porting process simple.</p>
<p>Let’s compare some data that I found interesting:</p>
<table>
<thead>
<tr>
<th>Metric</th>
<th style="text-align: right">Go</th>
<th style="text-align: right">Nim (clang backend)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compile command</td>
<td style="text-align: right"><code class="language-plaintext highlighter-rouge">go build</code></td>
<td style="text-align: right"><code class="language-plaintext highlighter-rouge">nimble build</code></td>
</tr>
<tr>
<td>Fresh compile time</td>
<td style="text-align: right">2.1 s¹</td>
<td style="text-align: right">1.7 s</td>
</tr>
<tr>
<td>Recompile time</td>
<td style="text-align: right">1.5 s</td>
<td style="text-align: right">0.5 s</td>
</tr>
<tr>
<td>Binary size</td>
<td style="text-align: right">16 MB (static)</td>
<td style="text-align: right">136 KB + 1MB SDL2</td>
</tr>
<tr>
<td>Source code size²</td>
<td style="text-align: right">3260 lines, 74 KB</td>
<td style="text-align: right">2145 lines, 60 KB</td>
</tr>
<tr>
<td>CPU usage</td>
<td style="text-align: right">71 %</td>
<td style="text-align: right">53 %</td>
</tr>
<tr>
<td>Memory usage</td>
<td style="text-align: right">79 MB</td>
<td style="text-align: right">73 MB</td>
</tr>
</tbody>
</table>
<p>¹ Excluding go-glfw, go-gl and portaudio, which take 17 s to compile<br />
² Emulation code only</p>
<p>It’s nice to see Nim doing well. Even the compile time is shorter than that of Go, which is well known for its short compile times. Now that the port seems to be doing fine and should be running on all Desktop platforms, let’s look into some other interesting things we can do with Nim:</p>
<h2 id="javascript-port-via-emscripten">JavaScript port via emscripten</h2>
<p>Nim has a JavaScript backend, but I don’t trust it to be stable enough for this task yet. So I opted for emscripten instead, which can compile C code into JavaScript. Since Nim outputs C code, this sounds like a perfect fit. Luckily eeeee helped me with getting it started, since he had experience by porting my <a href="https://ddnet.org/">DDNet</a> client to <a href="http://teewebs.net/">teewebs.net</a>.</p>
<p>It turned out that <a href="https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html">emsdk</a> is the easiest way to use emscripten:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./emsdk update
$ ./emsdk install latest
$ ./emsdk activate latest
$ source ./emsdk_env.sh
</code></pre></div></div>
<p>This may take a while, get a cup of tea. Afterwards we should have the <code class="language-plaintext highlighter-rouge">emconfigure</code>, <code class="language-plaintext highlighter-rouge">emmake</code> and <code class="language-plaintext highlighter-rouge">emcc</code> commands available. We can build regular Nim programs and look at the <a href="/nimes/hello.html">resulting html file</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat hello.nim
echo "Hello World"
$ nim --cc:clang --clang.exe:emcc --clang.linkerexe:emcc \
--cpu:i386 -d:release -o:hello.html c hello.nim
$ ls -lha hello.{html,js}
-rw-r--r-- 1 def users 101K Mai 1 19:02 hello.html
-rw-r--r-- 1 def users 385K Mai 1 19:02 hello.js
</code></pre></div></div>
<p>That’s a pretty cumbersome building command, so we’ll slim it down later. The next step is to <a href="https://hg.libsdl.org/SDL/file/e0e2e94ce5ea/docs/README-emscripten.md">build SDL2 for emscripten</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ hg clone https://hg.libsdl.org/SDL
$ cd SDL
$ emconfigure ./configure --host=asmjs-unknown-emscripten \
--disable-assembly --disable-threads \
--enable-cpuinfo=false CFLAGS="-O2"
$ emmake make
$ ls -lha build/.libs/libSDL2.a
-rw-r--r-- 1 def users 1.6M Apr 29 06:58 build/.libs/libSDL2.a
</code></pre></div></div>
<p>I put the resulting <code class="language-plaintext highlighter-rouge">libSDL2.a</code> into the NimES repository under <code class="language-plaintext highlighter-rouge">emscripten/</code> for convenience.</p>
<p>Instead of increasing the cumbersomeness of our build command anymore, NimES’s <a href="https://github.com/def-/nimes/blob/master/src/nim.cfg">nim.cfg</a> specifies how to compile when <code class="language-plaintext highlighter-rouge">-d:emscripten</code> is set:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@if emscripten:
define = SDL_Static
gc = none
cc = clang
clang.exe = "emcc"
clang.linkerexe = "emcc"
clang.options.linker = ""
cpu = "i386"
out = "nimes.html"
warning[GcMem] = off
passC = "-Wno-warn-absolute-paths -Iemscripten -s USE_SDL=2"
passL = "-O3 -Lemscripten -s USE_SDL=2 --preload-file tetris.nes --preload-file pacman.nes --preload-file smb.nes --preload-file smb3.nes -s TOTAL_MEMORY=16777216"
@end
</code></pre></div></div>
<p>Now a simple <code class="language-plaintext highlighter-rouge">nim -d:release -d:emscripten c src/nimes</code> builds the JavaScript port. Note that I’m preloading a few ROMs so that they can be loaded. The HTML then uses the <code class="language-plaintext highlighter-rouge">?nes=</code> parameter to pass the command line argument:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">var</span> <span class="nx">argument</span><span class="p">;</span>
<span class="nf">if </span><span class="p">(</span><span class="nx">QueryString</span><span class="p">.</span><span class="nf">hasOwnProperty</span><span class="p">(</span><span class="dl">"</span><span class="s2">nes</span><span class="dl">"</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">argument</span> <span class="o">=</span> <span class="nx">QueryString</span><span class="p">.</span><span class="nx">nes</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">argument</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">smb3.nes</span><span class="dl">"</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">Module</span><span class="p">;</span>
<span class="nx">Module</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">preRun</span><span class="p">:</span> <span class="p">[],</span>
<span class="na">postRun</span><span class="p">:</span> <span class="p">[],</span>
<span class="na">arguments</span><span class="p">:</span> <span class="p">[</span><span class="nx">argument</span><span class="p">],</span>
<span class="na">canvas</span><span class="p">:</span> <span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">canvas</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nf">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">canvas</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">canvas</span><span class="p">.</span><span class="nf">addEventListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">webglcontextlost</span><span class="dl">"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span> <span class="nf">alert</span><span class="p">(</span><span class="dl">'</span><span class="s1">WebGL context lost. You will need to reload the page.</span><span class="dl">'</span><span class="p">);</span> <span class="nx">e</span><span class="p">.</span><span class="nf">preventDefault</span><span class="p">();</span> <span class="p">},</span> <span class="kc">false</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">canvas</span><span class="p">;</span>
<span class="p">})(),</span>
<span class="na">totalDependencies</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">};</span></code></pre></figure>
<p>Inside the Nim source code there are some interesting changes too. I quickly wrapped these functions as there is no emscripten wrapper for Nim yet:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">when</span> <span class="n">defined</span><span class="p">(</span><span class="n">emscripten</span><span class="p">):</span>
<span class="k">proc </span><span class="nf">emscripten_set_main_loop</span><span class="p">(</span><span class="n">fun</span><span class="p">:</span> <span class="k">proc</span><span class="p">()</span> <span class="p">{.</span><span class="n">cdecl</span><span class="p">.},</span> <span class="n">fps</span><span class="p">,</span>
<span class="n">simulate_infinite_loop</span><span class="p">:</span> <span class="n">cint</span><span class="p">)</span> <span class="p">{.</span><span class="n">header</span><span class="p">:</span> <span class="s">"<emscripten.h>"</span><span class="p">.}</span>
<span class="k">proc </span><span class="nf">emscripten_cancel_main_loop</span><span class="p">()</span> <span class="p">{.</span><span class="n">header</span><span class="p">:</span> <span class="s">"<emscripten.h>"</span><span class="p">.}</span></code></pre></figure>
<p>Emscripten requires a slightly different execution style. Instead of actually looping, we define the main loop like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">when</span> <span class="n">defined</span><span class="p">(</span><span class="n">emscripten</span><span class="p">):</span>
<span class="n">emscripten_set_main_loop</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">while</span> <span class="n">runGame</span><span class="p">:</span>
<span class="n">loop</span><span class="p">()</span></code></pre></figure>
<p>That’s the main idea and with this we get a pretty playable <a href="/nimes/">web version of NimES</a>. I’m still getting 60fps in it, but just barely on my machine. Chrome seems to do a bit better than Firefox.</p>
<h2 id="android-port">Android port</h2>
<p>Obviously the next step is to port NimES to Android as well. But since the original emulator is more accurate and nice than performant, we shouldn’t expect runnable speed. Think of this more as a proof of concept:</p>
<p>We need a fresh clone of the SDL2 repository for this as well as the Android SDK (12 or later) and NDK (7 or later) installed. SDL2 has <a href="https://wiki.libsdl.org/Android">building instructions for Android</a> as well:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ hg clone https://hg.libsdl.org/SDL
$ cd SDL/build-scripts
$ ./androidbuild.sh org.nimes /dev/null
$ ls ../build/org.nimes
gen/ src/ build.properties local.properties
jni/ AndroidManifest.xml build.xml proguard-project.txt
res/ ant.properties default.properties project.properties
</code></pre></div></div>
<p>That’s our Android build directory now. I put this into the repository as well, under <code class="language-plaintext highlighter-rouge">android/</code>. Now we can add some ROM to the <code class="language-plaintext highlighter-rouge">assets/</code> directory and tell Nim to put the resulting C files into the correct directory and not to build them into binaries at all:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@if android:
cpu = "i386"
nimcache = "./android/jni/src"
compileOnly
noMain
@end
</code></pre></div></div>
<p>You may have noticed that I also defined <code class="language-plaintext highlighter-rouge">noMain</code>. Instead we define our own main function, as SDL is a bit weird with mains. Thanks to <a href="https://github.com/yglukhov">yglukhov</a> for this little trick:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">when</span> <span class="n">defined</span><span class="p">(</span><span class="n">android</span><span class="p">):</span>
<span class="p">{.</span><span class="n">emit</span><span class="p">:</span> <span class="s">"""
#include <SDL_main.h>
extern int cmdCount;
extern char** cmdLine;
extern char** gEnv;
N_CDECL(void, NimMain)(void);
int main(int argc, char** args) {
cmdLine = args;
cmdCount = argc;
gEnv = NULL;
NimMain();
return nim_program_result;
}
"""</span><span class="p">.}</span></code></pre></figure>
<p>Another trick is how to access the assets we embed into our APK. Luckily SDL2 provides functions for that, which we can use as replacements for the regular file operations:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">from</span> <span class="n">sdl2</span> <span class="k">import</span> <span class="n">rwFromFile</span><span class="p">,</span> <span class="n">read</span><span class="p">,</span> <span class="n">freeRW</span>
<span class="k">proc </span><span class="nf">newCartridge</span><span class="o">*</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="kt">string</span><span class="p">):</span> <span class="n">Cartridge</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">file</span> <span class="o">=</span> <span class="n">rwFromFile</span><span class="p">(</span><span class="n">path</span><span class="p">.</span><span class="n">cstring</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span>
<span class="k">defer</span><span class="p">:</span> <span class="n">freeRW</span> <span class="n">file</span>
<span class="k">var</span> <span class="n">header</span><span class="p">:</span> <span class="n">iNESHeader</span>
<span class="c"># Read directly into the header object</span>
<span class="k">if</span> <span class="n">read</span><span class="p">(</span><span class="n">file</span><span class="p">,</span> <span class="k">addr</span> <span class="n">header</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">sizeof</span> <span class="n">header</span><span class="p">)</span> <span class="o">!=</span> <span class="n">sizeof</span> <span class="n">header</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">newException</span><span class="p">(</span><span class="n">ValueError</span><span class="p">,</span> <span class="s">"header can't be read"</span><span class="p">)</span>
<span class="p">...</span></code></pre></figure>
<p>Finally we can build the project:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim -d:release -d:android c src/nimes
$ cd android
$ ndk-build
$ ant debug
</code></pre></div></div>
<p>And the end result is a nice <a href="/nimes/nimes.apk">nimes.apk</a>. Of course it only shows some low FPS video for now and doesn’t even have any controls, but it’s a start.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In the end I’m quite happy with the result: A truly portable emulator written in my favorite language. It compiles to C, C++ as well as JavaScript and runs on any Desktop platform as well as JavaScript and Android. The process for this was much easier than expected, mostly thanks to Nim and SDL2. I see a bright future for Nim as a practical language.</p>
<p>If you have any comments, suggestions or questions, feel free to ask them on <a href="https://news.ycombinator.com/item?id=9473653">Hacker News</a> or <a href="https://www.reddit.com/r/programming/comments/34jnv1/porting_a_nes_emulator_from_go_to_nim/">Reddit</a>.</p>
NimES: a NES emulator in Nim2015-04-30T00:00:00+02:00https://hookrace.net/blog/nimes
<p>Since today Nim 0.11 has been released, I guess it’s a good time to
release this as well: NimES is a NES emulator written in Nim. More info in the
<a href="https://github.com/def-/nimes">GitHub repository</a> and the <a href="/nimes/">JS demo</a>.</p>
<p>Discussion on <a href="https://news.ycombinator.com/item?id=9470580">Hacker News</a> and <a href="https://www.reddit.com/r/programming/comments/34gfah/nimes_nes_emulator_in_nim/">Reddit</a>.</p>
Make a Lisp in Nim2015-03-04T00:00:00+01:00https://hookrace.net/blog/make-a-lisp-in-nim
<p>I spent the last weekend working through the amazing
<a href="https://github.com/kanaka/mal/blob/master/process/guide.md">guide</a> for <a href="https://github.com/kanaka/mal">Make a
Lisp</a>, writing a Lisp interpreter in Nim. The
<a href="https://github.com/kanaka/mal/tree/master/nim">final result</a> just made it into
the repository.</p>
<p>Running the Nim version is pretty simple. You need the Nim compiler from the <a href="https://github.com/araq/nim">devel branch</a> and <a href="https://github.com/flaviut/nre">nre</a> which can be installed through <a href="https://github.com/nim-lang/nimble">nimble</a>. After installing those you can build and run the MAL interpreter:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd nim
$ make
# OR
$ nimble build
$ ./stepA_mal
Mal [nim]
user> 12
12
user> (+ 2 3)
5
</code></pre></div></div>
<!--more-->
<p>Running the tests:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ..
$ make "test^nim^step1" # A single test
$ make "test^nim" # All tests
$ make "perf^nim" # All benchmarks
</code></pre></div></div>
<p>There is a nice presentation in MAL about MAL you can read:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd nim
$ ./stepA_mal ../examples/clojurewest2014.mal
</code></pre></div></div>
<p>And you can run MAL implented in MAL itself using the Nim interpreter:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./stepA_mal ../mal/stepA_mal.mal
Mal [nim-mal]
mal-user>
</code></pre></div></div>
<p>The benchmark results from <a href="https://github.com/kanaka">Joel Martin</a>, the author
of MAL, don’t look bad for Nim. <em>Edit</em>: Note that these are just rough
measurements to see that the Nim implementation is doing fine. Don’t judge the
other languages for their numbers, which may not have ideal implementations
performance-wise.</p>
<p>In the short benchmarks Nim is the fastest, in the long benchmark only the JVM
can beat it. I didn’t really try to optimize and oriented mostly on the Python
implementation, which is quite a lot slower as you can see. So it’s pretty nice
to see idiomatic Nim performing well:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> perf1 perf2 perf3
macros math macros
ms ms iters/sec
java 6 24 17969
scala 47 87 15963
nim 1 1 11121
ocaml 1 3 7063
cs 10 11 5414
vb 12 13 4523
rust 2 5 4084
c 1 4 3649
go 1 6 3048
racket 3 10 2461
coffee 5 11 2326
js 6 14 1726
ruby 4 15 1255
haskell 4 14 1163
clojure 11 23 1174
forth 9 29 563
php 13 51 331
python 14 51 304
bash 1673 9000 276
perl 18 69 215
ps 41 332 48
R 116 490 28
miniMAL 779 3448 4
matlab 1688 5844 2
make 3427 28453 0
</code></pre></div></div>
<p>I also didn’t really aim for a low amount of code but rather good readability,
but it may still be interesting to look at. I guess Nim’s size being so close
to Python shows how much I oriented on the Python implementation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Lines Words Chars
mal 280 913 7075
clojure 369 1164 10099
ruby 481 1513 13247
coffee 532 2020 15653
racket 550 1937 17229
python 670 1997 19632
nim 715 2453 20263
r 812 2283 21644
lua 914 2640 23102
ocaml 597 3704 24371
scala 871 3060 24885
js 920 3059 26437
php 940 3011 27332
matlab 959 2485 28322
perl 1081 3511 29091
miniMAL 857 3104 29983
haskell 985 4452 30115
bash 1347 3586 31717
go 1297 4744 36321
ps 1409 6117 38651
forth 1609 6826 44715
cs 1249 4136 45039
java 1591 4966 53223
rust 1897 5702 56516
vb 1556 5175 58099
make 1593 6055 62310
c 2304 6957 73047
</code></pre></div></div>
<p>Thanks to Joel Martin for this guide. This was my first time getting close to
Lisp and I recommend others to work through the MAL guide as well. But be
aware, once you get through the first steps, it becomes addictive.</p>
<p>Comments on <a href="https://www.reddit.com/r/programming/comments/2xx4wq/make_a_lisp_in_nim/">Reddit</a> and <a href="https://news.ycombinator.com/item?id=9145360">Hacker News</a>.</p>
How I Start: Nim2015-02-09T00:00:00+01:00https://hookrace.net/blog/how-i-start-nim
<p>My most recent post goes in another direction (no, it’s still about Nim): How I
start a new Nim project. Check out <a href="http://howistart.org/posts/nim/1">the
article</a> on
<a href="http://howistart.org/">howistart.org</a>. Comments on <a href="https://news.ycombinator.com/item?id=9021244">Hacker
News</a> and
<a href="https://www.reddit.com/r/programming/comments/2vaxr9/how_i_start_nim_dennis_felsing/">Reddit</a> as always.</p>
Conclusion on Nim2015-01-26T00:00:00+01:00https://hookrace.net/blog/conclusion-on-nim
<p>In my last two posts, “<a href="/blog/what-is-special-about-nim/">What is special about Nim?</a>” and “<a href="/blog/what-makes-nim-practical/">What makes Nim practical</a>”, I forgot the important conclusion - why I personally have decided for Nim in favor of Rust, C++, Python and Haskell:</p>
<p>Nim is not the fastest language, it’s not the easiest language to write in and it surely has some flaws that should be fixed. Nim has no single “killer feature” like go’s goroutines or Rust’s memory management. But Nim doesn’t need a killer feature. Instead it strikes a reasonable balance that makes it the most efficient language for me:</p>
<ul>
<li>I can produce reasonably efficient code (faster than Python and Haskell)</li>
<li>that is reasonably readable (more so than Rust, C++ and Haskell)</li>
<li>in a reasonable amount of time (less than Rust, C++ and Haskell)</li>
</ul>
What makes Nim practical?2015-01-23T00:00:00+01:00https://hookrace.net/blog/what-makes-nim-practical
<p>In <a href="/blog/what-is-special-about-nim/">my last post</a> I showed what makes the <a href="http://nim-lang.org/">Nim</a> programming language special. Today, let’s consider Nim from another angle: What makes Nim a practical programming language?</p>
<h2 id="binary-distribution">Binary Distribution</h2>
<p>Programs written in interpreted languages like Python are difficult to distribute. Either you require Python (in a specific version even) to be installed already, or you ship it with your program. This even causes some to <a href="http://prog21.dadgum.com/203.html">reconsider Python as a teaching language</a>.</p>
<p>How does Nim work around this problem? For starters your program gets statically linked against the Nim runtime. That means you end up with a single binary that depends solely on the standard C library, which we can take for granted on any operating system we’re interested in.</p>
<!--more-->
<p>Let’s write a small program and give this a try:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="n">echo</span> <span class="s">"Hello World"</span></code></pre></figure>
<p>If you want to follow along, <a href="http://nim-lang.org/download.html">get the Nim compiler</a>. Save this code as <code class="language-plaintext highlighter-rouge">hello.nim</code>. Let’s compile it now so that we can distribute the <code class="language-plaintext highlighter-rouge">hello</code> binary:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim -d:release c hello
CC: hello
CC: stdlib_system
[Linking]
$ ./hello
Hello World
$ ls -lha hello
-rwxr-xr-x 1 def def 57K Jan 22 09:37 hello*
$ ldd hello
linux-vdso.so.1 (0x00007fffd5973000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f0f92c6b000)
libc.so.6 => /lib64/libc.so.6 (0x00007f0f928c3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0f92e6f000)
</code></pre></div></div>
<p>Now our little program starts growing and we get interested in using a few Nim libraries. Do we have to add compiled versions of these libraries to our distributions now? Do not fret! Nim libraries are statically compiled into our binary as well. Let’s get a library using Nim’s package manager, <a href="https://github.com/nim-lang/nimble">nimble</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nimble install strfmt
Installing strfmt-0.5.4
strfmt installed successfully.
</code></pre></div></div>
<p>And use the library in our, admittedly not very useful, program:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">strfmt</span>
<span class="n">echo</span> <span class="s">"Hello {} number {:04.1f}"</span><span class="p">.</span><span class="n">fmt</span><span class="p">(</span><span class="s">"World"</span><span class="p">,</span> <span class="mf">6.0</span><span class="p">)</span></code></pre></figure>
<p>If we want to compile with Clang instead of GCC as the backend, that’s easy as well and Clang is usually much faster to compile and yields a smaller binary:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --cc:clang -d:release c hello
$ ./hello
Hello World number 06.0
$ ls -lha hello
-rwxr-xr-x 1 def def 54K Jan 22 10:06 hello*
</code></pre></div></div>
<p>The only dynamic libraries we have to worry about are the C libraries we use, but that’s a story for another time.</p>
<h2 id="source-distribution">Source Distribution</h2>
<p>We can’t compile our program for all possible combinations of operating systems and CPU architectures of course. But we can be prepared for them and create a source distribution that can be compiled on the target systems with just a C compiler:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ tree
.
├── hello.ini
├── hello.nim
└── lib
└── nimbase.h
$ cat hello.ini
[Project]
Platforms: """
windows: i386;amd64
linux: i386;amd64;powerpc64;arm;sparc;mips;powerpc
macosx: i386;amd64;powerpc64
solaris: i386;amd64;sparc
freebsd: i386;amd64
netbsd: i386;amd64
openbsd: i386;amd64
haiku: i386;amd64
"""
[Lib]
Files: "lib/nimbase.h"
[Windows]
binPath: "bin"
$ niminst csource hello.ini -d:release
</code></pre></div></div>
<p>Here’s the resulting build script if you want to try it out on any of these platforms: <a href="/public/hello-csources.zip">hello-csources.zip</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ unzip hello-csources.zip
$ ./build.sh
SUCCESS
$ bin/hello
Hello World
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">niminst</code> can also be used to <a href="http://nim-lang.org/docs/niminst.html">build full installers</a> for your program. It’s what the Nim compiler itself uses.</p>
<h2 id="debug-and-release-builds">Debug and Release Builds</h2>
<p>Debug and release builds behave quite differently in Nim by default. Here’s an overview:</p>
<table>
<thead>
<tr>
<th>Option</th>
<th>Debug build</th>
<th>Release build</th>
</tr>
</thead>
<tbody>
<tr>
<td>objChecks</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>fieldChecks</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>rangeChecks</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>boundChecks</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>overflowChecks</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>assertions</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>stackTrace</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>lineTrace</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>lineDir</td>
<td>On</td>
<td>Off</td>
</tr>
<tr>
<td>deadCodeElim</td>
<td>Off</td>
<td>On</td>
</tr>
</tbody>
</table>
<p>As you develop your program you will be thankful for the range, bound and overflow checks. As you release your program your users will be thankful for the additional speed of disabling those.</p>
<p>If this is not what you want, and you require runtime checks even in release builds, you can enable them project wide or for parts of your code.</p>
<p>For example here we have an expensive max function that takes most of the running time:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">max</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">):</span> <span class="n">T</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">x</span><span class="p">.</span><span class="n">len</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">1</span> <span class="p">..</span> <span class="n">x</span><span class="p">.</span><span class="n">high</span><span class="p">:</span>
<span class="k">if</span> <span class="n">result</span> <span class="o"><</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span>
<span class="k">var</span> <span class="n">s</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="kt">int</span><span class="o">]</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">1_000</span><span class="p">:</span>
<span class="n">s</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="n">echo</span> <span class="n">max</span><span class="p">(</span><span class="n">s</span><span class="p">)</span></code></pre></figure>
<p>We want our program to never access out of sequence bounds, so we want to compile with <code class="language-plaintext highlighter-rouge">nim -d:release --checks:on max</code>. As it’s unreasonable to write this every time, we can create a <code class="language-plaintext highlighter-rouge">max.nim.cfg</code> file in our directory instead and enable runtime checks for every compilation of our program:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>checks: on
</code></pre></div></div>
<p>Now we run into the problem that the <code class="language-plaintext highlighter-rouge">max</code> function runs too slowly (it doesn’t, but let’s imagine). So we disable runtime checks just for this function:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="p">{.</span><span class="n">push</span> <span class="n">checks</span><span class="p">:</span> <span class="n">off</span><span class="p">.}</span>
<span class="k">proc </span><span class="nf">max</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">varargs</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">):</span> <span class="n">T</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">x</span><span class="p">.</span><span class="n">len</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">1</span> <span class="p">..</span> <span class="n">x</span><span class="p">.</span><span class="n">high</span><span class="p">:</span>
<span class="k">if</span> <span class="n">result</span> <span class="o"><</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">i</span><span class="o">]</span>
<span class="p">{.</span><span class="n">pop</span><span class="p">.}</span></code></pre></figure>
<p>Great, now we can control which part of our program has runtime checks and which doesn’t. <code class="language-plaintext highlighter-rouge">checks</code> enables and disables all runtime checks at once, but there are more <a href="http://nim-lang.org/docs/manual.html#pragmas-compilation-option-pragmas">fine-grained controls</a> as well.</p>
<h2 id="nim-instead-of-python--c">Nim instead of Python + C++</h2>
<p>High performance languages like C++ may require some boilerplate. A higher level language can be used at compile time to automatically create the boilerplate. In <a href="https://ddnet.org/">DDNet</a> (which inherited them from <a href="https://www.teeworlds.com/">Teeworlds</a>) there are <a href="https://github.com/def-/ddnet/tree/DDRace64/datasrc">many examples</a> for this.</p>
<p>A really simple use case is to get the current git revision at compile time in Python and put it into the config.h with a <code class="language-plaintext highlighter-rouge">#define</code> so it can be referred to at runtime in the program. In Nim you need neither Python nor a C preprocessor and can instead do it all directly at compile time in Nim:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">const</span> <span class="n">gitHash</span> <span class="o">=</span> <span class="n">staticExec</span> <span class="s">"git rev-parse HEAD"</span>
<span class="n">echo</span> <span class="s">"Revision: "</span><span class="p">,</span> <span class="n">gitHash</span></code></pre></figure>
<h2 id="use-as-a-scripting-language">Use as a scripting language</h2>
<p>With TinyCC as the backend Nim makes for a nice scripting language. All we need is a <code class="language-plaintext highlighter-rouge">nimscript</code> file and the TinyCC compiler:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/sh
nim --cc:tcc --verbosity:0 -d:release -r c $*
</code></pre></div></div>
<p>To automatically update DDNet’s map files on our upcoming HTTP map server I’m using this script (with this <a href="https://github.com/def-/nim-unsorted/blob/master/crc32.nim">crc32</a> module):</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="c">#!/usr/bin/env nimscript</span>
<span class="k">import</span> <span class="n">os</span><span class="p">,</span> <span class="n">crc32</span><span class="p">,</span> <span class="n">strutils</span>
<span class="k">const</span>
<span class="n">baseDir</span> <span class="o">=</span> <span class="s">"/home/teeworlds/servers"</span>
<span class="n">mapdlDir</span> <span class="o">=</span> <span class="s">"/var/www-maps"</span>
<span class="k">for</span> <span class="n">kind</span><span class="p">,</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">walkDir</span> <span class="n">baseDir</span><span class="o">/</span><span class="s">"maps"</span><span class="p">:</span>
<span class="k">if</span> <span class="n">kind</span> <span class="o">!=</span> <span class="n">pcFile</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">let</span> <span class="p">(</span><span class="n">dir</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">ext</span><span class="p">)</span> <span class="o">=</span> <span class="n">splitFile</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ext</span> <span class="o">!=</span> <span class="s">".map"</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">let</span>
<span class="n">sum</span> <span class="o">=</span> <span class="n">crc32FromFile</span><span class="p">(</span><span class="n">path</span><span class="p">).</span><span class="kt">int64</span><span class="p">.</span><span class="n">toHex</span><span class="p">(</span><span class="mi">8</span><span class="p">).</span><span class="n">toLower</span>
<span class="n">newName</span> <span class="o">=</span> <span class="n">name</span> <span class="o">&</span> <span class="s">"_"</span> <span class="o">&</span> <span class="n">sum</span> <span class="o">&</span> <span class="n">ext</span>
<span class="n">newPath</span> <span class="o">=</span> <span class="n">mapdlDir</span> <span class="o">/</span> <span class="n">newName</span>
<span class="n">tmpPath</span> <span class="o">=</span> <span class="n">newPath</span> <span class="o">&</span> <span class="s">".tmp"</span>
<span class="k">if</span> <span class="n">existsFile</span> <span class="n">newPath</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">copyFile</span> <span class="n">path</span><span class="p">,</span> <span class="n">tmpPath</span>
<span class="n">moveFile</span> <span class="n">tmpPath</span><span class="p">,</span> <span class="n">newPath</span></code></pre></figure>
<p>Of course you can always compile your program if you need the full speed of Clang or GCC. But for quick tests while developing, or scripts you just need occasionally, this is good enough.</p>
<p>There is a nicer version of this idea now, called <a href="https://github.com/flaviut/nimrun">nimrun</a></p>
<h2 id="debugging-nim">Debugging Nim</h2>
<p>GDB works rather well with Nim. The only problem I’ve had so far is that the variables get unique identifiers in C, but using tab completion you can get along fine:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="n">echo</span> <span class="s">"Hello World"</span>
<span class="k">var</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">echo</span> <span class="s">"Value of x: "</span><span class="p">,</span> <span class="n">x</span></code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nim --lineDir:on --debuginfo c hello
$ gdb ./hello
(gdb) break hello.nim:3
Breakpoint 1 at 0x41f886: file /home/def/hello.nim, line 3.
(gdb) run
Starting program: /home/def/hello
Hello World
Breakpoint 1, helloInit () at /home/def/hello.nim:3
3 echo "Value of x: ", x
(gdb) print x_89010
$1 = 10
(gdb) print x_89010 = 200
$2 = 200
(gdb) c
Continuing.
Value of x: 200
</code></pre></div></div>
<h2 id="wrapping-libraries-with-c2nim">Wrapping libraries with c2nim</h2>
<p>Nice C libraries can automatically be converted into a Nim wrapper with <a href="https://github.com/nim-lang/c2nim">c2nim</a>. Let’s do this for Bellard’s new <a href="http://bellard.org/bpg/">BPG image format</a>. Since the library is so new, there is no shared library compilation included yet, so I <a href="https://github.com/def-/libbpg/commit/6253393fd70c318022db9f920c2d41c5f70b6c34">added that</a> myself in <a href="https://github.com/def-/libbpg">my fork</a>.</p>
<p>We need to fix up the C header for c2nim by adding this at the top of <a href="https://github.com/def-/libbpg/blob/master/libbpg.h">libbpg.h</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#ifdef C2NIM
# dynlib bpglib
# cdecl
/* Dynamically link to the correct library for our system: */
# if defined(windows)
# define bpglib "libbpg.dll"
# elif defined(macosx)
# define bpglib "libbpg.dylib"
# else
# define bpglib "libbpg.so"
# endif
/* Remove prefixes in our wrapper, we have modules in Nim: */
# prefix bpg_decoder_
# prefix BPG_
# prefix BPG
/* These are not recognized by c2nim, but that's easy to fix: */
# mangle uint8_t uint8
# mangle uint16_t uint16
# mangle uint32_t uint32
#endif
</code></pre></div></div>
<p>I made a <a href="https://github.com/def-/nim-bpg/blob/master/libbpg.h.patch">patch</a> so I can apply the changes again when the libbpg interface changes. Creating the wrapper is now as simple as this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ patch -p1 < libbpg.h.patch
patching file libbpg.h
$ c2nim -o:bpg.nim libbpg.h
Hint: operation successful (155 lines compiled; 0 sec total; 516.528KB) [SuccessX]
</code></pre></div></div>
<p>Look, no hands! The <a href="https://github.com/def-/nim-bpg/blob/master/src/bpg.nim">resulting wrapper</a> doesn’t even look bad. Now we can write a simple program to decode a BPG file and save it in the PPM format:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">bpg</span><span class="p">,</span> <span class="n">os</span>
<span class="k">proc </span><span class="nf">writePPM</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">imgInfo</span><span class="p">:</span> <span class="n">ImageInfo</span>
<span class="k">discard</span> <span class="n">img</span><span class="p">.</span><span class="n">getInfo</span><span class="p">(</span><span class="k">addr</span> <span class="n">imgInfo</span><span class="p">)</span>
<span class="k">let</span> <span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="n">h</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="n">imgInfo</span><span class="p">.</span><span class="n">width</span><span class="p">.</span><span class="kt">int</span><span class="p">,</span> <span class="n">imgInfo</span><span class="p">.</span><span class="n">height</span><span class="p">.</span><span class="kt">int</span><span class="p">)</span>
<span class="k">var</span> <span class="n">rgbLine</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="n">uint8</span><span class="o">]</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="mi">3</span><span class="p">)</span>
<span class="k">var</span> <span class="n">f</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">fmWrite</span><span class="p">)</span>
<span class="n">f</span><span class="p">.</span><span class="n">writeln</span> <span class="s">"P6</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="s">" "</span><span class="p">,</span> <span class="n">h</span><span class="p">,</span> <span class="s">"</span><span class="se">\n</span><span class="s">255"</span>
<span class="k">discard</span> <span class="n">img</span><span class="p">.</span><span class="n">start</span><span class="p">(</span><span class="n">OUTPUT_FORMAT_RGB24</span><span class="p">)</span>
<span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="n">h</span><span class="p">:</span>
<span class="k">discard</span> <span class="n">img</span><span class="p">.</span><span class="n">getLine</span><span class="p">(</span><span class="k">addr</span> <span class="n">rgbLine</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">)</span>
<span class="k">discard</span> <span class="n">f</span><span class="p">.</span><span class="n">writeBuffer</span><span class="p">(</span><span class="k">addr</span> <span class="n">rgbLine</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">,</span> <span class="n">w</span> <span class="o">*</span> <span class="mi">3</span><span class="p">)</span>
<span class="n">f</span><span class="p">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">if</span> <span class="n">paramCount</span><span class="p">()</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">stderr</span><span class="p">.</span><span class="n">writeln</span> <span class="s">"Usage: decode img.bpg"</span>
<span class="n">quit</span> <span class="mi">1</span>
<span class="k">var</span>
<span class="n">buf</span> <span class="o">=</span> <span class="n">readFile</span> <span class="n">paramStr</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">img</span> <span class="o">=</span> <span class="n">bpg</span><span class="p">.</span><span class="n">open</span><span class="p">()</span>
<span class="k">if</span> <span class="n">img</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="k">cast</span><span class="o">[</span><span class="k">ptr</span> <span class="n">uint8</span><span class="o">]</span><span class="p">(</span><span class="k">addr</span> <span class="n">buf</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">),</span> <span class="n">buf</span><span class="p">.</span><span class="n">len</span><span class="p">.</span><span class="n">cint</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">stderr</span><span class="p">.</span><span class="n">writeln</span> <span class="s">"Could not decode image"</span>
<span class="n">quit</span> <span class="mi">2</span>
<span class="n">img</span><span class="p">.</span><span class="n">writePPM</span><span class="p">(</span><span class="s">"out.ppm"</span><span class="p">)</span>
<span class="n">img</span><span class="p">.</span><span class="n">close</span><span class="p">()</span></code></pre></figure>
<p>Note that you can tell from the <code class="language-plaintext highlighter-rouge">discard</code> statements where I chose to ignore errors. If you actually want to run this, check out the <a href="https://github.com/def-/nim-bpg">instructions in the repository</a>.</p>
<h2 id="final-words">Final words</h2>
<p>That’s Nim from a more practical angle. Hopefully you’ll consider Nim for your next project, there are <a href="http://nim-lang.org/docs/lib.html">many libraries</a> available already. Also, the community always needs more helping hands!</p>
<p>To do my part in making Nim more practical, I’m trying to implement a new, working REPL using TinyCC. We may also soon get a working compiler as a service for proper IDE integration, directly from Andreas Rumpf.</p>
<p>For comments use <a href="https://www.reddit.com/r/programming/comments/2tedjb/what_makes_nim_practical/">Reddit</a>, <a href="https://news.ycombinator.com/item?id=8934582">Hacker News</a> or get in touch with the <a href="http://nim-lang.org/community.html">Nim community</a> directly. You can reach me personally at <a href="mailto:dennis@felsing.org">dennis@felsing.org</a>.</p>
<p>Thanks to Andreas Rumpf and Dominik Picheta again, for proof-reading this post as well.</p>
What is special about Nim?2015-01-01T00:00:00+01:00https://hookrace.net/blog/what-is-special-about-nim
<p><a href="http://habrahabr.ru/post/258119/">Russian Translation by frol</a>, <a href="https://segmentfault.com/a/1190000002576013">Chinese Translation by JiyinYiyong</a>, <a href="https://qiita.com/momeemt/items/0a8dca5eb43d9b8cfbe9">Japanese Translation by Mutsuha Asada</a></p>
<p>The <a href="http://nim-lang.org/">Nim programming language</a> is exciting. While the <a href="http://nim-lang.org/docs/tut1.html">official tutorial</a> is great, it slowly introduces you to the language. Instead I want to quickly show what you can do with Nim that would be more difficult or impossible in other languages.</p>
<p>I discovered Nim in my quest to find the right tools to write a game, HookRace, the successor of my current <a href="https://ddnet.org">DDNet</a> game/mod of Teeworlds. Since I’m busy with other projects for now, this blog is now officially about Nim instead, until I find time to continue developing the game.</p>
<h2 id="easy-to-get-running">Easy to get running</h2>
<p>Ok, this part is not exciting yet, but I invite you to follow along with the post:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">0</span><span class="p">..</span><span class="mi">10</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Hello World"</span><span class="o">[</span><span class="mf">0</span><span class="p">..</span><span class="n">i</span><span class="o">]</span></code></pre></figure>
<!--more-->
<p>If you want to do so, <a href="http://nim-lang.org/download.html">get the Nim compiler</a>. Save this code as <code class="language-plaintext highlighter-rouge">hello.nim</code>, compile it with <code class="language-plaintext highlighter-rouge">nim c hello</code> and finally run the binary with <code class="language-plaintext highlighter-rouge">./hello</code>. To immediately compile and run, use <code class="language-plaintext highlighter-rouge">nim -r c hello</code>. To use an optimized release build instead of a debug build use <code class="language-plaintext highlighter-rouge">nim -d:release c hello</code>. With all of these settings you will see the following output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>H
He
Hel
Hell
Hello
Hello
Hello W
Hello Wo
Hello Wor
Hello Worl
Hello World
</code></pre></div></div>
<h2 id="run-regular-code-at-compile-time">Run regular code at compile time</h2>
<p>To implement an efficient CRC32 procedure you need a lookup table. You could compute it at runtime or write it into your code as a magic array. Clearly we don’t want any magic numbers in our code, so we’ll do it at runtime (for now):</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">unsigned</span><span class="p">,</span> <span class="n">strutils</span>
<span class="k">type</span> <span class="n">CRC32</span><span class="o">*</span> <span class="o">=</span> <span class="n">uint32</span>
<span class="k">const</span> <span class="n">initCRC32</span><span class="o">*</span> <span class="o">=</span> <span class="n">CRC32</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="k">proc </span><span class="nf">createCRCTable</span><span class="p">():</span> <span class="kt">array</span><span class="o">[</span><span class="mi">256</span><span class="p">,</span> <span class="n">CRC32</span><span class="o">]</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">0</span><span class="p">..</span><span class="mi">255</span><span class="p">:</span>
<span class="k">var</span> <span class="n">rem</span> <span class="o">=</span> <span class="n">CRC32</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="mf">0</span><span class="p">..</span><span class="mi">7</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">rem</span> <span class="ow">and</span> <span class="mi">1</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> <span class="n">rem</span> <span class="o">=</span> <span class="p">(</span><span class="n">rem</span> <span class="ow">shr</span> <span class="mi">1</span><span class="p">)</span> <span class="ow">xor</span> <span class="n">CRC32</span><span class="p">(</span><span class="mh">0xedb88320</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span> <span class="n">rem</span> <span class="o">=</span> <span class="n">rem</span> <span class="ow">shr</span> <span class="mi">1</span>
<span class="n">result</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">=</span> <span class="n">rem</span>
<span class="c"># Table created at runtime</span>
<span class="k">var</span> <span class="n">crc32table</span> <span class="o">=</span> <span class="n">createCRCTable</span><span class="p">()</span>
<span class="k">proc </span><span class="nf">crc32</span><span class="p">(</span><span class="n">s</span><span class="p">):</span> <span class="n">CRC32</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">initCRC32</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">(</span><span class="n">result</span> <span class="ow">shr</span> <span class="mi">8</span><span class="p">)</span> <span class="ow">xor</span> <span class="n">crc32table</span><span class="o">[</span><span class="p">(</span><span class="n">result</span> <span class="ow">and</span> <span class="mh">0xff</span><span class="p">)</span> <span class="ow">xor</span> <span class="n">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)</span><span class="o">]</span>
<span class="n">result</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">result</span>
<span class="c"># String conversion proc $, automatically called by echo</span>
<span class="k">proc </span><span class="nf">`$`</span><span class="p">(</span><span class="n">c</span><span class="p">:</span> <span class="n">CRC32</span><span class="p">):</span> <span class="kt">string</span> <span class="o">=</span> <span class="kt">int64</span><span class="p">(</span><span class="n">c</span><span class="p">).</span><span class="n">toHex</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
<span class="n">echo</span> <span class="n">crc32</span><span class="p">(</span><span class="s">"The quick brown fox jumps over the lazy dog"</span><span class="p">)</span></code></pre></figure>
<p>Great, this works and we get <code class="language-plaintext highlighter-rouge">414FA339</code> as the output. But it would be even better if we could compute the CRC table at compile time. This is as easy as it gets in Nim, instead of the current crc32table creation we use:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="c"># Table created at compile time</span>
<span class="k">const</span> <span class="n">crc32table</span> <span class="o">=</span> <span class="n">createCRCTable</span><span class="p">()</span></code></pre></figure>
<p>Yes, that’s right: All we had to do was replace the <code class="language-plaintext highlighter-rouge">var</code> with a <code class="language-plaintext highlighter-rouge">const</code>. Beautiful, isn’t it? We can write the exact same code and have it run at runtime and compile time. No template metaprogramming necessary.</p>
<h2 id="extend-the-language">Extend the language</h2>
<p><a href="http://nim-lang.org/docs/manual.html#templates">Templates</a> and <a href="http://nim-lang.org/docs/manual.html#macros">macros</a> can be used to get rid of boilerplate, by transforming your code at compile time.</p>
<p>Templates just replace their invocations with their code at compile-time. We can define our own loops like this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">times</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="n">expr</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">stmt</span><span class="p">):</span> <span class="n">stmt</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="n">x</span><span class="p">:</span>
<span class="n">y</span>
<span class="mf">10.</span><span class="n">times</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Hello World"</span></code></pre></figure>
<p>So the compiler transforms the times-loop to this regular for-loop:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">10</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Hello World"</span></code></pre></figure>
<p>If you’re curious about the <code class="language-plaintext highlighter-rouge">10.times:</code> syntax, it’s just a regular call to <code class="language-plaintext highlighter-rouge">times</code> with <code class="language-plaintext highlighter-rouge">10</code> as the first parameter and the following indented block as the second parameter. Instead you could also write <code class="language-plaintext highlighter-rouge">times(10):</code>, see <a href="#unified-call-syntax">Unified Call Syntax</a>.</p>
<p>Or initialize sequences (variable sized arrays) more comfortably:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">newSeqWith</span><span class="p">(</span><span class="n">len</span><span class="p">:</span> <span class="kt">int</span><span class="p">,</span> <span class="n">init</span><span class="p">:</span> <span class="n">expr</span><span class="p">):</span> <span class="n">expr</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">result</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="k">type</span><span class="p">(</span><span class="n">init</span><span class="p">)</span><span class="o">]</span><span class="p">(</span><span class="n">len</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">0</span> <span class="p">..</span> <span class="o"><</span><span class="n">len</span><span class="p">:</span>
<span class="n">result</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">=</span> <span class="n">init</span>
<span class="n">result</span>
<span class="c"># Create a 2-dimensional sequence of size 20,10</span>
<span class="k">var</span> <span class="n">seq2D</span> <span class="o">=</span> <span class="n">newSeqWith</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="n">newSeq</span><span class="o">[</span><span class="kt">bool</span><span class="o">]</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
<span class="k">import</span> <span class="n">math</span>
<span class="n">randomize</span><span class="p">()</span>
<span class="c"># Create a sequence of 20 random integers smaller than 10</span>
<span class="k">var</span> <span class="n">seqRand</span> <span class="o">=</span> <span class="n">newSeqWith</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="n">random</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
<span class="n">echo</span> <span class="n">seqRand</span></code></pre></figure>
<p>Macros go a step further and allow you to analyze and manipulate the AST. There are no list comprehensions in Nim, for example, but it’s possible to <a href="https://github.com/def-/nimrod-unsorted/blob/master/listcomprehensions.nim">add them to the language by using a macro</a>. Now instead of this:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">res</span><span class="p">:</span> <span class="kt">seq</span><span class="o">[</span><span class="kt">int</span><span class="o">]</span> <span class="o">=</span> <span class="o">@[]</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">10</span><span class="p">:</span>
<span class="k">if</span> <span class="n">x</span> <span class="ow">mod</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">res</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">echo</span> <span class="n">res</span>
<span class="k">const</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">20</span>
<span class="k">var</span> <span class="n">result</span><span class="p">:</span> <span class="kt">seq</span><span class="o">[</span><span class="k">tuple</span><span class="o">[</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">,</span><span class="n">c</span><span class="p">:</span> <span class="kt">int</span><span class="o">]]</span> <span class="o">=</span> <span class="o">@[]</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="n">n</span><span class="p">:</span>
<span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="n">x</span><span class="p">..</span><span class="n">n</span><span class="p">:</span>
<span class="k">for</span> <span class="n">z</span> <span class="ow">in</span> <span class="n">y</span><span class="p">..</span><span class="n">n</span><span class="p">:</span>
<span class="k">if</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="o">*</span><span class="n">y</span> <span class="o">==</span> <span class="n">z</span><span class="o">*</span><span class="n">z</span><span class="p">:</span>
<span class="n">result</span><span class="p">.</span><span class="n">add</span><span class="p">((</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">z</span><span class="p">))</span>
<span class="n">echo</span> <span class="n">result</span></code></pre></figure>
<p>You can use the <a href="http://nim-lang.org/docs/future.html">future module</a> and write:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">future</span>
<span class="n">echo</span> <span class="n">lc</span><span class="o">[</span><span class="n">x</span> <span class="o">|</span> <span class="p">(</span><span class="n">x</span> <span class="o"><-</span> <span class="mf">1</span><span class="p">..</span><span class="mi">10</span><span class="p">,</span> <span class="n">x</span> <span class="ow">mod</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">),</span> <span class="kt">int</span><span class="o">]</span>
<span class="k">const</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">20</span>
<span class="n">echo</span> <span class="n">lc</span><span class="o">[</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">z</span><span class="p">)</span> <span class="o">|</span> <span class="p">(</span><span class="n">x</span> <span class="o"><-</span> <span class="mf">1</span><span class="p">..</span><span class="n">n</span><span class="p">,</span> <span class="n">y</span> <span class="o"><-</span> <span class="n">x</span><span class="p">..</span><span class="n">n</span><span class="p">,</span> <span class="n">z</span> <span class="o"><-</span> <span class="n">y</span><span class="p">..</span><span class="n">n</span><span class="p">,</span>
<span class="n">x</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="o">*</span><span class="n">y</span> <span class="o">==</span> <span class="n">z</span><span class="o">*</span><span class="n">z</span><span class="p">),</span> <span class="k">tuple</span><span class="o">[</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">,</span><span class="n">c</span><span class="p">:</span> <span class="kt">int</span><span class="o">]]</span></code></pre></figure>
<h2 id="add-your-own-optimizations-to-the-compiler">Add your own optimizations to the compiler</h2>
<p>Instead of optimizing your code, wouldn’t you prefer to make the compiler smarter? In Nim you can!</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">x</span><span class="p">:</span> <span class="kt">int</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mf">1</span><span class="p">..</span><span class="mi">1_000_000_000</span><span class="p">:</span>
<span class="n">x</span> <span class="o">+=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">i</span>
<span class="n">echo</span> <span class="n">x</span></code></pre></figure>
<p>This (pretty useless) code can be sped up by teaching the compiler two optimizations:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">optMul</span><span class="p">{`</span><span class="o">*</span><span class="p">`(</span><span class="n">a</span><span class="p">,</span><span class="mi">2</span><span class="p">)}(</span><span class="n">a</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="kt">int</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">x</span> <span class="o">=</span> <span class="n">a</span>
<span class="n">x</span> <span class="o">+</span> <span class="n">x</span>
<span class="k">template</span> <span class="n">canonMul</span><span class="p">{`</span><span class="o">*</span><span class="p">`(</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">)}(</span><span class="n">a</span><span class="p">:</span> <span class="kt">int</span><span class="p">{</span><span class="n">lit</span><span class="p">},</span> <span class="n">b</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="kt">int</span> <span class="o">=</span>
<span class="n">b</span> <span class="o">*</span> <span class="n">a</span></code></pre></figure>
<p>In the first <a href="http://nim-lang.org/docs/manual.html#term-rewriting-macros">term rewriting</a> template we specify that <code class="language-plaintext highlighter-rouge">a * 2</code> can be replaced by <code class="language-plaintext highlighter-rouge">a + a</code>. In the second one we specify that <code class="language-plaintext highlighter-rouge">int</code>s in a multiplication can be swapped if the first is an integer literal, so that we can potentially apply the first template.</p>
<p>More complicated patterns can also be implemented, for example to optimize boolean logic:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">template</span> <span class="n">optLog1</span><span class="p">{</span><span class="n">a</span> <span class="ow">and</span> <span class="n">a</span><span class="p">}(</span><span class="n">a</span><span class="p">):</span> <span class="n">auto</span> <span class="o">=</span> <span class="n">a</span>
<span class="k">template</span> <span class="n">optLog2</span><span class="p">{</span><span class="n">a</span> <span class="ow">and</span> <span class="p">(</span><span class="n">b</span> <span class="ow">or</span> <span class="p">(</span><span class="ow">not</span> <span class="n">b</span><span class="p">))}(</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">):</span> <span class="n">auto</span> <span class="o">=</span> <span class="n">a</span>
<span class="k">template</span> <span class="n">optLog3</span><span class="p">{</span><span class="n">a</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">a</span><span class="p">}(</span><span class="n">a</span><span class="p">:</span> <span class="kt">int</span><span class="p">):</span> <span class="n">auto</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">var</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">12</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">x</span> <span class="ow">and</span> <span class="n">x</span>
<span class="c"># Hint: optLog1(x) --> ’x’ [Pattern]</span>
<span class="n">r</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span> <span class="ow">and</span> <span class="n">x</span><span class="p">)</span> <span class="ow">and</span> <span class="p">((</span><span class="n">s</span> <span class="ow">or</span> <span class="n">s</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="ow">not</span> <span class="p">(</span><span class="n">s</span> <span class="ow">or</span> <span class="n">s</span><span class="p">)))</span>
<span class="c"># Hint: optLog2(x and x, s or s) --> ’x and x’ [Pattern]</span>
<span class="c"># Hint: optLog1(x) --> ’x’ [Pattern]</span>
<span class="n">q</span> <span class="o">=</span> <span class="p">(</span><span class="n">s</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">x</span><span class="p">)</span> <span class="ow">and</span> <span class="ow">not</span> <span class="p">(</span><span class="n">s</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">x</span><span class="p">)</span>
<span class="c"># Hint: optLog3(s and not x) --> ’0’ [Pattern]</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">s</code> gets optimized to <code class="language-plaintext highlighter-rouge">x</code> directly, <code class="language-plaintext highlighter-rouge">r</code> gets optimized to <code class="language-plaintext highlighter-rouge">x</code> in 2 successive pattern applications and <code class="language-plaintext highlighter-rouge">q</code> ends up as <code class="language-plaintext highlighter-rouge">0</code> immediately.</p>
<p>If you want to see how term rewriting templates can be used to avoid allocations with bigints, look for the templates starting with <code class="language-plaintext highlighter-rouge">opt</code> in the <a href="https://github.com/def-/nim-bigints/blob/master/src/bigints.nim">bigints library</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">bigints</span>
<span class="k">var</span> <span class="n">i</span> <span class="o">=</span> <span class="mf">0.</span><span class="n">initBigInt</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">echo</span> <span class="n">i</span></code></pre></figure>
<h2 id="bind-to-your-favorite-c-functions-and-libraries">Bind to your favorite C functions and libraries</h2>
<p>Since Nim compiles down to C, <a href="http://nim-lang.org/docs/manual.html#foreign-function-interface">foreign function interfaces</a> are fun.</p>
<p>You can easily use your favorite functions from the C standard library:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">printf</span><span class="p">(</span><span class="n">formatstr</span><span class="p">:</span> <span class="n">cstring</span><span class="p">)</span>
<span class="p">{.</span><span class="n">header</span><span class="p">:</span> <span class="s">"<stdio.h>"</span><span class="p">,</span> <span class="n">varargs</span><span class="p">.}</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%s %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="s">"foo"</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span></code></pre></figure>
<p>Or use your own code written in C:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">hi</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">)</span> <span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"awesome %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">name</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="p">{.</span><span class="n">compile</span><span class="p">:</span> <span class="s">"hi.c"</span><span class="p">.}</span>
<span class="k">proc </span><span class="nf">hi</span><span class="o">*</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="n">cstring</span><span class="p">)</span> <span class="p">{.</span><span class="n">importc</span><span class="p">.}</span>
<span class="n">hi</span> <span class="s">"from Nim"</span></code></pre></figure>
<p>Or any library you please with the help of <a href="https://github.com/nim-lang/c2nim">c2nim</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">set_default_dpi</span><span class="o">*</span><span class="p">(</span><span class="n">dpi</span><span class="p">:</span> <span class="n">cdouble</span><span class="p">)</span> <span class="p">{.</span><span class="n">cdecl</span><span class="p">,</span>
<span class="n">importc</span><span class="p">:</span> <span class="s">"rsvg_set_default_dpi"</span><span class="p">,</span>
<span class="n">dynlib</span><span class="p">:</span> <span class="s">"librsvg-2.so"</span><span class="p">.}</span></code></pre></figure>
<h2 id="control-the-garbage-collector">Control the garbage collector</h2>
<p>To achieve soft realtime, you can tell the <a href="http://nim-lang.org/docs/gc.html">garbage collector</a> when and for how long it is allowed to run. The main game logic can be implemented like this in Nim, to prevent the garbage collector from causing stutters:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="n">gcDisable</span><span class="p">()</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="n">gameLogic</span><span class="p">()</span>
<span class="n">renderFrame</span><span class="p">()</span>
<span class="n">gcStep</span><span class="p">(</span><span class="n">us</span> <span class="o">=</span> <span class="n">leftTime</span><span class="p">)</span>
<span class="n">sleep</span><span class="p">(</span><span class="n">restTime</span><span class="p">)</span></code></pre></figure>
<h2 id="type-safe-sets-and-arrays-of-enums">Type safe sets and arrays of enums</h2>
<p>Often you want a mathematical set over values you defined yourself. Here’s how you do this in type-safe way:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">type</span> <span class="n">FakeTune</span> <span class="o">=</span> <span class="k">enum</span>
<span class="n">freeze</span><span class="p">,</span> <span class="n">solo</span><span class="p">,</span> <span class="n">noJump</span><span class="p">,</span> <span class="n">noColl</span><span class="p">,</span> <span class="n">noHook</span><span class="p">,</span> <span class="n">jetpack</span>
<span class="k">var</span> <span class="n">x</span><span class="p">:</span> <span class="kt">set</span><span class="o">[</span><span class="n">FakeTune</span><span class="o">]</span>
<span class="n">x</span><span class="p">.</span><span class="n">incl</span> <span class="n">freeze</span>
<span class="n">x</span><span class="p">.</span><span class="n">incl</span> <span class="n">solo</span>
<span class="n">x</span><span class="p">.</span><span class="n">excl</span> <span class="n">solo</span>
<span class="n">echo</span> <span class="n">x</span> <span class="o">+</span> <span class="p">{</span><span class="n">noColl</span><span class="p">,</span> <span class="n">noHook</span><span class="p">}</span>
<span class="k">if</span> <span class="n">freeze</span> <span class="ow">in</span> <span class="n">x</span><span class="p">:</span>
<span class="n">echo</span> <span class="s">"Here be freeze"</span>
<span class="k">var</span> <span class="n">y</span> <span class="o">=</span> <span class="p">{</span><span class="n">solo</span><span class="p">,</span> <span class="n">noHook</span><span class="p">}</span>
<span class="n">y</span><span class="p">.</span><span class="n">incl</span> <span class="mi">0</span> <span class="c"># Error: type mismatch</span></code></pre></figure>
<p>You can’t accidentally add a value of another type. Internally the set works as an efficient bitvector.</p>
<p>The same is possible with arrays, indexing them with your enum:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">a</span><span class="p">:</span> <span class="kt">array</span><span class="o">[</span><span class="n">FakeTune</span><span class="p">,</span> <span class="kt">int</span><span class="o">]</span>
<span class="n">a</span><span class="o">[</span><span class="n">freeze</span><span class="o">]</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">echo</span> <span class="n">a</span><span class="o">[</span><span class="n">freeze</span><span class="o">]</span></code></pre></figure>
<h2 id="unified-call-syntax">Unified Call Syntax</h2>
<p>This is just syntactic sugar, but it’s definitely nice to have. In Python I always forget whether <code class="language-plaintext highlighter-rouge">len</code> and <code class="language-plaintext highlighter-rouge">append</code> are functions or methods. In Nim I don’t have to remember, because there is no difference. Nim uses <a href="http://nim-lang.org/docs/manual.html#procedures-method-call-syntax">Unified Call Syntax</a>, which has now also been proposed for C++ by <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4165.pdf">Herb Sutter</a> and <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4174.pdf">Bjarne Stroustrup</a>.</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">var</span> <span class="n">xs</span> <span class="o">=</span> <span class="o">@[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="o">]</span>
<span class="c"># Procedure call syntax</span>
<span class="n">add</span><span class="p">(</span><span class="n">xs</span><span class="p">,</span> <span class="mi">4_000_000</span><span class="p">)</span>
<span class="n">echo</span> <span class="n">len</span><span class="p">(</span><span class="n">xs</span><span class="p">)</span>
<span class="c"># Method call syntax</span>
<span class="n">xs</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="m">0b0101_0000_0000</span><span class="p">)</span>
<span class="n">echo</span> <span class="n">xs</span><span class="p">.</span><span class="n">len</span><span class="p">()</span>
<span class="c"># Command invocation syntax</span>
<span class="n">xs</span><span class="p">.</span><span class="n">add</span> <span class="mh">0x06_FF_FF_FF</span>
<span class="n">echo</span> <span class="n">xs</span><span class="p">.</span><span class="n">len</span></code></pre></figure>
<h2 id="good-performance">Good performance</h2>
<p>It’s easy to write fast code in Nim, as can be seen in the <a href="https://github.com/logicchains/LPATHBench/blob/master/writeup.md">Longest Path Finding Benchmark</a>, in which Nim is competing with <a href="https://github.com/logicchains/LPATHBench/blob/master/nim.nim">some cute code</a>.</p>
<p>I made some measurements on my machine when the benchmark was first published (Linux x86-64, Intel Core2Quad Q9300 @2.5GHz, state of 2014-12-20):</p>
<table>
<thead>
<tr>
<th>Lang</th>
<th style="text-align: right">Time [ms]</th>
<th style="text-align: right">Memory [KB]</th>
<th style="text-align: right">Compile Time [ms]</th>
<th style="text-align: right">Compressed Code [B]</th>
</tr>
</thead>
<tbody>
<tr>
<td>Nim</td>
<td style="text-align: right">1400</td>
<td style="text-align: right">1460</td>
<td style="text-align: right">893</td>
<td style="text-align: right">486</td>
</tr>
<tr>
<td>C++</td>
<td style="text-align: right">1478</td>
<td style="text-align: right">2717</td>
<td style="text-align: right">774</td>
<td style="text-align: right">728</td>
</tr>
<tr>
<td>D</td>
<td style="text-align: right">1518</td>
<td style="text-align: right">2388</td>
<td style="text-align: right">1614</td>
<td style="text-align: right">669</td>
</tr>
<tr>
<td>Rust</td>
<td style="text-align: right">1623</td>
<td style="text-align: right">2632</td>
<td style="text-align: right">6735</td>
<td style="text-align: right">934</td>
</tr>
<tr>
<td>Java</td>
<td style="text-align: right">1874</td>
<td style="text-align: right">24428</td>
<td style="text-align: right">812</td>
<td style="text-align: right">778</td>
</tr>
<tr>
<td>OCaml</td>
<td style="text-align: right">2384</td>
<td style="text-align: right">4496</td>
<td style="text-align: right">125</td>
<td style="text-align: right">782</td>
</tr>
<tr>
<td>Go</td>
<td style="text-align: right">3116</td>
<td style="text-align: right">1664</td>
<td style="text-align: right">596</td>
<td style="text-align: right">618</td>
</tr>
<tr>
<td>Haskell</td>
<td style="text-align: right">3329</td>
<td style="text-align: right">5268</td>
<td style="text-align: right">3002</td>
<td style="text-align: right">1091</td>
</tr>
<tr>
<td>LuaJit</td>
<td style="text-align: right">3857</td>
<td style="text-align: right">2368</td>
<td style="text-align: right">-</td>
<td style="text-align: right">519</td>
</tr>
<tr>
<td>Lisp</td>
<td style="text-align: right">8219</td>
<td style="text-align: right">15876</td>
<td style="text-align: right">1043</td>
<td style="text-align: right">1007</td>
</tr>
<tr>
<td>Racket</td>
<td style="text-align: right">8503</td>
<td style="text-align: right">130284</td>
<td style="text-align: right">24793</td>
<td style="text-align: right">741</td>
</tr>
</tbody>
</table>
<p>Compressed code size with <code class="language-plaintext highlighter-rouge">gzip -9 < nim.nim | wc -c</code>. Removed unused functions in Haskell. Compile times are clean compiles, if you have a nimcache with the standard library precompiled it’s only 323 ms for Nim.</p>
<p>Another small benchmark I did, calculating which numbers of the first 100 million are prime, in Python, Nim and C:</p>
<h3 id="python-runtime-351-s">Python (runtime: 35.1 s)</h3>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">eratosthenes</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="n">sieve</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">*</span> <span class="mi">2</span> <span class="o">+</span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="nf">int</span><span class="p">(</span><span class="n">n</span><span class="o">**</span><span class="mf">0.5</span><span class="p">)):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">sieve</span><span class="p">[</span><span class="n">i</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">i</span><span class="o">*</span><span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
<span class="n">sieve</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">sieve</span>
<span class="nf">eratosthenes</span><span class="p">(</span><span class="mi">100000000</span><span class="p">)</span></code></pre></figure>
<h3 id="nim-runtime-26-s">Nim (runtime: 2.6 s)</h3>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">math</span>
<span class="k">proc </span><span class="nf">eratosthenes</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="n">auto</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="kt">int8</span><span class="o">]</span><span class="p">(</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="n">result</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="n">result</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">0</span> <span class="p">..</span> <span class="kt">int</span> <span class="n">sqrt</span><span class="p">(</span><span class="kt">float</span> <span class="n">n</span><span class="p">):</span>
<span class="k">if</span> <span class="n">result</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">countup</span><span class="p">(</span><span class="n">i</span><span class="o">*</span><span class="n">i</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
<span class="n">result</span><span class="o">[</span><span class="n">j</span><span class="o">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">discard</span> <span class="n">eratosthenes</span><span class="p">(</span><span class="mi">100_000_000</span><span class="p">)</span></code></pre></figure>
<h3 id="c-runtime-26-s">C (runtime: 2.6 s)</h3>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="cp">#include</span> <span class="cpf"><stdlib.h></span><span class="cp">
#include</span> <span class="cpf"><math.h></span><span class="cp">
</span><span class="kt">char</span><span class="o">*</span> <span class="nf">eratosthenes</span><span class="p">(</span><span class="kt">int</span> <span class="n">n</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">char</span><span class="o">*</span> <span class="n">sieve</span> <span class="o">=</span> <span class="n">calloc</span><span class="p">(</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span><span class="k">sizeof</span><span class="p">(</span><span class="kt">char</span><span class="p">));</span>
<span class="n">sieve</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="n">sieve</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">m</span> <span class="o">=</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span> <span class="n">sqrt</span><span class="p">((</span><span class="kt">double</span><span class="p">)</span> <span class="n">n</span><span class="p">);</span>
<span class="k">for</span><span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><=</span> <span class="n">m</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">sieve</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="o">*</span><span class="n">i</span><span class="p">;</span> <span class="n">j</span> <span class="o"><=</span> <span class="n">n</span><span class="p">;</span> <span class="n">j</span> <span class="o">+=</span> <span class="n">i</span><span class="p">)</span>
<span class="n">sieve</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">sieve</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">eratosthenes</span><span class="p">(</span><span class="mi">100000000</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<h2 id="compile-to-javascript">Compile to JavaScript</h2>
<p>You can <a href="http://nim-lang.org/docs/backends.html#backends-the-javascript-target">compile Nim to JavaScript</a>, instead of C. This allows you to write the client as well as the server directly in Nim. Let’s make a small visitor counter on the server that gets displayed in the browser. This is our <code class="language-plaintext highlighter-rouge">client.nim</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">htmlgen</span><span class="p">,</span> <span class="n">dom</span>
<span class="k">type</span> <span class="n">Data</span> <span class="o">=</span> <span class="k">object</span>
<span class="n">visitors</span> <span class="p">{.</span><span class="n">importc</span><span class="p">.}:</span> <span class="kt">int</span>
<span class="n">uniques</span> <span class="p">{.</span><span class="n">importc</span><span class="p">.}:</span> <span class="kt">int</span>
<span class="n">ip</span> <span class="p">{.</span><span class="n">importc</span><span class="p">.}:</span> <span class="n">cstring</span>
<span class="k">proc </span><span class="nf">printInfo</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="n">Data</span><span class="p">)</span> <span class="p">{.</span><span class="n">exportc</span><span class="p">.}</span> <span class="o">=</span>
<span class="k">var</span> <span class="n">infoDiv</span> <span class="o">=</span> <span class="n">document</span><span class="p">.</span><span class="n">getElementById</span><span class="p">(</span><span class="s">"info"</span><span class="p">)</span>
<span class="n">infoDiv</span><span class="p">.</span><span class="n">innerHTML</span> <span class="o">=</span> <span class="n">p</span><span class="p">(</span><span class="s">"You're visitor number "</span><span class="p">,</span> <span class="o">$</span><span class="n">data</span><span class="p">.</span><span class="n">visitors</span><span class="p">,</span>
<span class="s">", unique visitor number "</span><span class="p">,</span> <span class="o">$</span><span class="n">data</span><span class="p">.</span><span class="n">uniques</span><span class="p">,</span>
<span class="s">" today. Your IP is "</span><span class="p">,</span> <span class="o">$</span><span class="n">data</span><span class="p">.</span><span class="n">ip</span><span class="p">,</span> <span class="s">"."</span><span class="p">)</span></code></pre></figure>
<p>We define a <code class="language-plaintext highlighter-rouge">Data</code> type that we use to pass data from the server to client. The <code class="language-plaintext highlighter-rouge">printInfo</code> procedure will be called with this <code class="language-plaintext highlighter-rouge">data</code> and display it. Compile this with <code class="language-plaintext highlighter-rouge">nim js client</code>. The result javascript file ends up in <code class="language-plaintext highlighter-rouge">nimcache/client.js</code>.</p>
<p>For the server we need to <a href="https://github.com/nim-lang/nimble">get the Nimble package manager</a> and run <code class="language-plaintext highlighter-rouge">nimble install jester</code>. Now we can use the <a href="https://github.com/dom96/jester">Jester web framework</a> and write our <code class="language-plaintext highlighter-rouge">server.nim</code>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">jester</span><span class="p">,</span> <span class="n">asyncdispatch</span><span class="p">,</span> <span class="n">json</span><span class="p">,</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">times</span><span class="p">,</span> <span class="n">sets</span><span class="p">,</span> <span class="n">htmlgen</span><span class="p">,</span> <span class="n">strtabs</span><span class="p">,</span> <span class="n">httpcore</span>
<span class="k">var</span>
<span class="n">visitors</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">uniques</span> <span class="o">=</span> <span class="n">initSet</span><span class="o">[</span><span class="kt">string</span><span class="o">]</span><span class="p">()</span>
<span class="n">time</span><span class="p">:</span> <span class="n">TimeInfo</span>
<span class="n">routes</span><span class="p">:</span>
<span class="n">get</span> <span class="s">"/"</span><span class="p">:</span>
<span class="n">resp</span> <span class="n">body</span><span class="p">(</span>
<span class="p">`</span><span class="ow">div</span><span class="p">`(</span><span class="n">id</span><span class="o">=</span><span class="s">"info"</span><span class="p">),</span>
<span class="n">script</span><span class="p">(</span><span class="n">src</span><span class="o">=</span><span class="s">"/client.js"</span><span class="p">,</span> <span class="p">`</span><span class="k">type</span><span class="p">`</span><span class="o">=</span><span class="s">"text/javascript"</span><span class="p">),</span>
<span class="n">script</span><span class="p">(</span><span class="n">src</span><span class="o">=</span><span class="s">"/visitors"</span><span class="p">,</span> <span class="p">`</span><span class="k">type</span><span class="p">`</span><span class="o">=</span><span class="s">"text/javascript"</span><span class="p">))</span>
<span class="n">get</span> <span class="s">"/client.js"</span><span class="p">:</span>
<span class="k">const</span> <span class="n">result</span> <span class="o">=</span> <span class="n">staticExec</span> <span class="s">"nim -d:release js client"</span>
<span class="k">const</span> <span class="n">clientJS</span> <span class="o">=</span> <span class="n">staticRead</span> <span class="s">"nimcache/client.js"</span>
<span class="n">resp</span> <span class="n">clientJS</span>
<span class="n">get</span> <span class="s">"/visitors"</span><span class="p">:</span>
<span class="k">let</span> <span class="n">newTime</span> <span class="o">=</span> <span class="n">getTime</span><span class="p">().</span><span class="n">getLocalTime</span>
<span class="k">if</span> <span class="n">newTime</span><span class="p">.</span><span class="n">monthDay</span> <span class="o">!=</span> <span class="n">time</span><span class="p">.</span><span class="n">monthDay</span><span class="p">:</span>
<span class="n">visitors</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">init</span> <span class="n">uniques</span>
<span class="n">time</span> <span class="o">=</span> <span class="n">newTime</span>
<span class="n">inc</span> <span class="n">visitors</span>
<span class="k">let</span> <span class="n">ip</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">request</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="n">hasKey</span> <span class="s">"X-Forwarded-For"</span><span class="p">:</span>
<span class="n">request</span><span class="p">.</span><span class="n">headers</span><span class="o">[</span><span class="s">"X-Forwarded-For"</span><span class="p">,</span> <span class="mi">0</span><span class="o">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">request</span><span class="p">.</span><span class="n">ip</span>
<span class="n">uniques</span><span class="p">.</span><span class="n">incl</span> <span class="n">ip</span>
<span class="k">let</span> <span class="n">json</span> <span class="o">=</span> <span class="o">%</span><span class="p">{</span><span class="s">"visitors"</span><span class="p">:</span> <span class="o">%</span><span class="n">visitors</span><span class="p">,</span>
<span class="s">"uniques"</span><span class="p">:</span> <span class="o">%</span><span class="n">uniques</span><span class="p">.</span><span class="n">len</span><span class="p">,</span>
<span class="s">"ip"</span><span class="p">:</span> <span class="o">%</span><span class="n">ip</span><span class="p">}</span>
<span class="n">resp</span> <span class="s">"printInfo(</span><span class="si">$#</span><span class="s">)"</span><span class="p">.</span><span class="n">format</span><span class="p">(</span><span class="n">json</span><span class="p">)</span>
<span class="n">runForever</span><span class="p">()</span></code></pre></figure>
<p>The server delivers the main website. It also delivers the <code class="language-plaintext highlighter-rouge">client.js</code>, by compiling and reading the <code class="language-plaintext highlighter-rouge">client.nim</code> at compile time. The logic is in the <code class="language-plaintext highlighter-rouge">/visitors</code> handling. Compile and run with <code class="language-plaintext highlighter-rouge">nim -r c server</code> and open <a href="http://localhost:5000/">http://localhost:5000/</a> to see the code in effect.</p>
<p>You can see our code in action on the <a href="/visitors/visitors">Jester-generated site</a> or inline here:</p>
<div id="info"><p>You don't have JavaScript enabled or something went wrong.</p></div>
<script src="/visitors/client.js" type="text/javascript"></script>
<script src="/visitors/visitors" type="text/javascript"></script>
<h2 id="final-words">Final words</h2>
<p>I hope I could pique your interest in in the Nim programming language.</p>
<p>Note that the language is not completely stable yet. Especially with the more obscure features you may run into bugs. But on the bright side, Nim 1.0 is supposed to be released within the next 3 months! So now is the perfect time to get started with learning Nim.</p>
<p>Bonus: Since Nim compiles to C and only depends on the C standard library you can deploy it nearly everywhere, including x86-64, ARM and Intel Xeon Phi accelerator cards.</p>
<p>For comments use <a href="https://www.reddit.com/r/programming/comments/2r06ej/what_is_special_about_nim/">Reddit</a>, <a href="https://news.ycombinator.com/item?id=8822816">Hacker News</a> or ask the Nim community directly on IRC (<a href="irc://irc.freenode.net/nim">#nim on freenode</a>). You can reach me personally at <a href="mailto:dennis@felsing.org">dennis@felsing.org</a>.</p>
<p>Thanks to Andreas Rumpf and Dominik Picheta for proof-reading this post.</p>
Nim Adventures2014-07-13T00:00:00+02:00https://hookrace.net/blog/nim-adventures
<p>To learn some <a href="http://nim-lang.org/">Nim</a> I held a talk about it at the <a href="https://entropia.de/GPN14">GPN14</a> (in German, <a href="https://dennis.felsin.org/nimrod/nimrod-gpn14.pdf">Slides</a>).</p>
<p>Afterwards I started solving <a href="http://rosettacode.org/wiki/Rosetta_Code">Rosetta Code</a> tasks in <a href="http://rosettacode.org/wiki/Category:Nimrod">Nim</a> to get a better feel for the language and standard library. That also made me discover some <a href="https://github.com/Araq/Nimrod/issues/created_by/def-?page=1&state=open">rough edges</a> in the language, but luckily the community, albeit small, is active and competent. All the small code pieces I wrote are now on <a href="https://github.com/search?q=user%3Adef-+nim">Github</a> too.</p>
<p>What I noticed is that most problems are as easy to solve as in Python, but much more performant. I’m now more confident that this language is the right choice for writing HookRace in.</p>
<!--more-->
<p>Some highlights:</p>
<h2 id="insertion-sort"><a href="http://rosettacode.org/wiki/Sorting_algorithms/Insertion_sort#Nimrod">Insertion Sort</a></h2>
<p>Once in Python:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">insertion_sort</span><span class="p">(</span><span class="n">l</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nf">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nf">len</span><span class="p">(</span><span class="n">l</span><span class="p">)):</span>
<span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="o">-</span><span class="mi">1</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">l</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="nf">while </span><span class="p">(</span><span class="n">l</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="o">></span> <span class="n">key</span><span class="p">)</span> <span class="ow">and</span> <span class="p">(</span><span class="n">j</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">):</span>
<span class="n">l</span><span class="p">[</span><span class="n">j</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">l</span><span class="p">[</span><span class="n">j</span><span class="p">]</span>
<span class="n">j</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="n">l</span><span class="p">[</span><span class="n">j</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">key</span></code></pre></figure>
<p>and in Nim:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">insertSort</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="k">var</span> <span class="n">openarray</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">)</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">1</span> <span class="p">..</span> <span class="o"><</span><span class="n">a</span><span class="p">.</span><span class="n">len</span><span class="p">:</span>
<span class="k">let</span> <span class="n">value</span> <span class="o">=</span> <span class="n">a</span><span class="o">[</span><span class="n">i</span><span class="o">]</span>
<span class="k">var</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span>
<span class="k">while</span> <span class="n">j</span> <span class="o">></span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">value</span> <span class="o"><</span> <span class="n">a</span><span class="o">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="o">]</span><span class="p">:</span>
<span class="n">a</span><span class="o">[</span><span class="n">j</span><span class="o">]</span> <span class="o">=</span> <span class="n">a</span><span class="o">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="o">]</span>
<span class="n">dec</span> <span class="n">j</span>
<span class="n">a</span><span class="o">[</span><span class="n">j</span><span class="o">]</span> <span class="o">=</span> <span class="n">value</span></code></pre></figure>
<h2 id="rank-languages-by-popularity"><a href="http://rosettacode.org/wiki/Rosetta_Code/Rank_languages_by_popularity#Nimrod">Rank languages by popularity</a></h2>
<p>The standard library is extremely useful, providing an <a href="http://nim-lang.org/docs/httpclient.html">HTTP client</a>, <a href="http://nim-lang.org/docs/json.html">JSON parser</a>, <a href="http://nim-lang.org/docs/re.html">regular expressions</a>, <a href="http://nim-lang.org/docs/strutils.html">string utils</a> and <a href="http://nim-lang.org/docs/algorithm.html">algorithms</a>, which I use to create a ranking of the popularity of languages on Rosetta Code:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">import</span> <span class="n">httpclient</span><span class="p">,</span> <span class="n">json</span><span class="p">,</span> <span class="n">re</span><span class="p">,</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">algorithm</span><span class="p">,</span> <span class="n">future</span>
<span class="k">const</span>
<span class="n">langSite</span> <span class="o">=</span> <span class="s">"http://www.rosettacode.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Programming_Languages&cmlimit=500&format=json"</span>
<span class="n">catSize</span> <span class="o">=</span> <span class="s">"http://www.rosettacode.org/w/index.php?title=Special:Categories&limit=5000"</span>
<span class="k">let</span> <span class="n">regex</span> <span class="o">=</span> <span class="s">re"title=</span><span class="se">"</span><span class="s">"Category:(.*?)"">.+?</a>.*\((.*) members\)"</span>
<span class="k">var</span> <span class="n">langs</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="kt">string</span><span class="o">]</span><span class="p">()</span>
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">parseJson</span><span class="p">(</span><span class="n">getContent</span><span class="p">(</span><span class="n">langSite</span><span class="p">))</span><span class="o">[</span><span class="s">"query"</span><span class="o">][</span><span class="s">"categorymembers"</span><span class="o">]</span><span class="p">:</span>
<span class="n">langs</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">l</span><span class="o">[</span><span class="s">"title"</span><span class="o">]</span><span class="p">.</span><span class="n">str</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"Category:"</span><span class="p">)</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span><span class="p">)</span>
<span class="k">var</span> <span class="n">ranks</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="k">tuple</span><span class="o">[</span><span class="n">lang</span><span class="p">:</span> <span class="kt">string</span><span class="p">,</span> <span class="n">count</span><span class="p">:</span> <span class="kt">int</span><span class="o">]]</span><span class="p">()</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">getContent</span><span class="p">(</span><span class="n">catSize</span><span class="p">).</span><span class="n">findAll</span><span class="p">(</span><span class="n">regex</span><span class="p">):</span>
<span class="k">let</span> <span class="n">lang</span> <span class="o">=</span> <span class="n">line</span><span class="p">.</span><span class="n">replacef</span><span class="p">(</span><span class="n">regex</span><span class="p">,</span> <span class="s">"</span><span class="si">$1</span><span class="s">"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">lang</span> <span class="ow">in</span> <span class="n">langs</span><span class="p">:</span>
<span class="k">let</span> <span class="n">count</span> <span class="o">=</span> <span class="n">parseInt</span><span class="p">(</span><span class="n">line</span><span class="p">.</span><span class="n">replacef</span><span class="p">(</span><span class="n">regex</span><span class="p">,</span> <span class="s">"</span><span class="si">$2</span><span class="s">"</span><span class="p">).</span><span class="n">strip</span><span class="p">())</span>
<span class="n">ranks</span><span class="p">.</span><span class="n">add</span><span class="p">((</span><span class="n">lang</span><span class="p">,</span> <span class="n">count</span><span class="p">))</span>
<span class="n">ranks</span><span class="p">.</span><span class="n">sort</span><span class="p">((</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span> <span class="o">=></span> <span class="n">cmp</span><span class="o">[</span><span class="kt">int</span><span class="o">]</span><span class="p">(</span><span class="n">y</span><span class="p">.</span><span class="n">count</span><span class="p">,</span> <span class="n">x</span><span class="p">.</span><span class="n">count</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">ranks</span><span class="p">:</span>
<span class="n">echo</span> <span class="n">align</span><span class="p">(</span><span class="o">$</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">),</span> <span class="mi">3</span><span class="p">),</span> <span class="n">align</span><span class="p">(</span><span class="o">$</span><span class="n">l</span><span class="p">.</span><span class="n">count</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> <span class="s">" - "</span><span class="p">,</span> <span class="n">l</span><span class="p">.</span><span class="n">lang</span></code></pre></figure>
<h2 id="calling-a-c-function"><a href="http://rosettacode.org/wiki/Call_a_foreign-language_function#Nimrod">Calling a C function</a></h2>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">proc </span><span class="nf">printf</span><span class="p">(</span><span class="n">formatstr</span><span class="p">:</span> <span class="n">cstring</span><span class="p">)</span> <span class="p">{.</span><span class="n">header</span><span class="p">:</span> <span class="s">"<stdio.h>"</span><span class="p">,</span> <span class="n">varargs</span><span class="p">.}</span>
<span class="k">var</span> <span class="n">x</span> <span class="o">=</span> <span class="s">"foo"</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"Hello %d %s!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span></code></pre></figure>
<h2 id="wrapping-a-shared-c-library"><a href="http://rosettacode.org/wiki/Call_a_function_in_a_shared_library#Nimrod">Wrapping a shared C library</a></h2>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">when</span> <span class="n">defined</span><span class="p">(</span><span class="n">windows</span><span class="p">):</span>
<span class="k">const</span> <span class="n">imglib</span> <span class="o">=</span> <span class="s">"imglib.dll"</span>
<span class="k">elif</span> <span class="n">defined</span><span class="p">(</span><span class="n">macosx</span><span class="p">):</span>
<span class="k">const</span> <span class="n">imglib</span> <span class="o">=</span> <span class="s">"imglib.dylib"</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">const</span> <span class="n">imglib</span> <span class="o">=</span> <span class="s">"imglib.so"</span>
<span class="k">proc </span><span class="nf">openimage</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="n">cstring</span><span class="p">):</span> <span class="n">cint</span> <span class="p">{.</span><span class="n">importc</span><span class="p">,</span> <span class="n">dynlib</span><span class="p">:</span> <span class="n">imglib</span><span class="p">.}</span>
<span class="n">echo</span> <span class="n">openimage</span><span class="p">(</span><span class="s">"foo"</span><span class="p">)</span></code></pre></figure>
<h2 id="quine"><a href="http://rosettacode.org/wiki/Quine#Nimrod">Quine</a></h2>
<p>A quine is a program that prints its own source code. This Nim program does that once at compiletime and once at runtime:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="k">const</span> <span class="n">x</span> <span class="o">=</span> <span class="s">"const x = |const y = x[0..9]&34.chr&x&34.chr&10.chr&x[11..100]&10.chr&x[102..115]&10.chr&x[117 .. -1]|static: echo y|echo y"</span>
<span class="k">const</span> <span class="n">y</span> <span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="mf">0</span><span class="p">..</span><span class="mi">9</span><span class="o">]&</span><span class="mf">34.</span><span class="n">chr</span><span class="o">&</span><span class="n">x</span><span class="o">&</span><span class="mf">34.</span><span class="n">chr</span><span class="o">&</span><span class="mf">10.</span><span class="n">chr</span><span class="o">&</span><span class="n">x</span><span class="o">[</span><span class="mf">11</span><span class="p">..</span><span class="mi">100</span><span class="o">]&</span><span class="mf">10.</span><span class="n">chr</span><span class="o">&</span><span class="n">x</span><span class="o">[</span><span class="mf">102</span><span class="p">..</span><span class="mi">115</span><span class="o">]&</span><span class="mf">10.</span><span class="n">chr</span><span class="o">&</span><span class="n">x</span><span class="o">[</span><span class="mi">117</span> <span class="p">..</span> <span class="o">-</span><span class="mi">1</span><span class="o">]</span>
<span class="k">static</span><span class="p">:</span> <span class="n">echo</span> <span class="n">y</span>
<span class="n">echo</span> <span class="n">y</span></code></pre></figure>
What is HookRace?2014-07-07T00:00:00+02:00https://hookrace.net/blog/what-is-hookrace
<p>I have been running <a href="https://ddnet.org">DDraceNetwork</a> for 1 year, which started out as a little server of a <a href="https://www.teeworlds.com">Teeworlds</a> modification called DDRace. It has turned into the biggest project within Teeworlds, offering servers in 6 countries with thousands of players and dozens of mappers. We also offer a custom server and client (Windows, Linux, OS X, Android).</p>
<p>Now it’s time to start something new. I’m working on a new game called HookRace, which will be the successor to DDraceNetwork. It will be its own game, not based on Teeworlds.</p>
<p>Some of the plans I have so far for HookRace:</p>
<!--more-->
<ul>
<li>Vector Graphics (SVG) ingame for a sharp image on any screen</li>
<li>Whole physics is executed at the client as well as the server, for perfect prediction</li>
<li>No lags with high player numbers</li>
<li>Accounts to prevent faking</li>
</ul>
<p>As the programming language for HookRace, I chose <a href="http://nim-lang.org">Nim</a>:</p>
<figure class="highlight"><pre><code class="language-nimrod" data-lang="nimrod"><span class="c"># Game of Life in terminal</span>
<span class="k">import</span> <span class="n">os</span><span class="p">,</span> <span class="n">strutils</span><span class="p">,</span> <span class="n">math</span>
<span class="n">randomize</span><span class="p">()</span>
<span class="k">var</span> <span class="n">w</span><span class="p">,</span> <span class="n">h</span><span class="p">:</span> <span class="kt">int</span>
<span class="k">if</span> <span class="n">paramCount</span><span class="p">()</span> <span class="o">>=</span> <span class="mi">2</span><span class="p">:</span>
<span class="n">w</span> <span class="o">=</span> <span class="n">parseInt</span> <span class="n">paramStr</span> <span class="mi">1</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">parseInt</span> <span class="n">paramStr</span> <span class="mi">2</span>
<span class="k">if</span> <span class="n">w</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">:</span> <span class="n">w</span> <span class="o">=</span> <span class="mi">30</span>
<span class="k">if</span> <span class="n">h</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">:</span> <span class="n">h</span> <span class="o">=</span> <span class="mi">30</span>
<span class="c"># Iterate over fields in the universe</span>
<span class="k">iterator</span> <span class="n">fields</span><span class="p">(</span><span class="n">a</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">),</span> <span class="n">b</span> <span class="o">=</span> <span class="p">(</span><span class="n">h</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">w</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span> <span class="o">=</span>
<span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="n">a</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">..</span><span class="n">b</span><span class="o">[</span><span class="mi">0</span><span class="o">]</span><span class="p">:</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">a</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span><span class="p">..</span><span class="n">b</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span><span class="p">:</span>
<span class="k">yield</span> <span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">x</span><span class="p">)</span>
<span class="c"># Create a sequence with an initializer</span>
<span class="k">proc </span><span class="nf">newSeqWith</span><span class="o">[</span><span class="n">T</span><span class="o">]</span><span class="p">(</span><span class="n">len</span><span class="p">:</span> <span class="kt">int</span><span class="p">,</span> <span class="n">init</span><span class="p">:</span> <span class="n">T</span><span class="p">):</span> <span class="kt">seq</span><span class="o">[</span><span class="n">T</span><span class="o">]</span> <span class="o">=</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">newSeq</span><span class="o">[</span><span class="n">T</span><span class="o">]</span> <span class="n">len</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="mi">0</span> <span class="p">..</span> <span class="o"><</span><span class="n">len</span><span class="p">:</span>
<span class="n">result</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">=</span> <span class="n">init</span>
<span class="c"># Initialize</span>
<span class="k">var</span> <span class="n">univ</span><span class="p">,</span> <span class="n">univNew</span> <span class="o">=</span> <span class="n">newSeqWith</span><span class="p">(</span><span class="n">h</span><span class="p">,</span> <span class="n">newSeq</span><span class="o">[</span><span class="kt">bool</span><span class="o">]</span> <span class="n">w</span><span class="p">)</span>
<span class="k">for</span> <span class="n">y</span><span class="p">,</span><span class="n">x</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">():</span>
<span class="k">if</span> <span class="n">random</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="o"><</span> <span class="mi">1</span><span class="p">:</span> <span class="n">univ</span><span class="o">[</span><span class="n">y</span><span class="o">][</span><span class="n">x</span><span class="o">]</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">while</span> <span class="kp">true</span><span class="p">:</span>
<span class="c"># Show</span>
<span class="n">stdout</span><span class="p">.</span><span class="n">write</span> <span class="s">"</span><span class="se">\e</span><span class="s">[H"</span>
<span class="k">for</span> <span class="n">y</span><span class="p">,</span><span class="n">x</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">():</span>
<span class="n">stdout</span><span class="p">.</span><span class="n">write</span> <span class="k">if</span> <span class="n">univ</span><span class="o">[</span><span class="n">y</span><span class="o">][</span><span class="n">x</span><span class="o">]</span><span class="p">:</span> <span class="s">"</span><span class="se">\e</span><span class="s">[07m </span><span class="se">\e</span><span class="s">[m"</span> <span class="k">else</span><span class="p">:</span> <span class="s">" "</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">stdout</span><span class="p">.</span><span class="n">write</span> <span class="s">"</span><span class="se">\e</span><span class="s">[E"</span>
<span class="n">stdout</span><span class="p">.</span><span class="n">flushFile</span>
<span class="c"># Evolve</span>
<span class="k">for</span> <span class="n">y</span><span class="p">,</span><span class="n">x</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">():</span>
<span class="k">var</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">y1</span><span class="p">,</span><span class="n">x1</span> <span class="ow">in</span> <span class="n">fields</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="p">)):</span>
<span class="k">if</span> <span class="n">univ</span><span class="o">[</span><span class="p">(</span><span class="n">y1</span><span class="o">+</span><span class="n">h</span><span class="p">)</span> <span class="ow">mod</span> <span class="n">h</span><span class="o">][</span><span class="p">(</span><span class="n">x1</span> <span class="o">+</span> <span class="n">w</span><span class="p">)</span> <span class="ow">mod</span> <span class="n">w</span><span class="o">]</span><span class="p">:</span>
<span class="n">inc</span> <span class="n">n</span>
<span class="k">if</span> <span class="n">univ</span><span class="o">[</span><span class="n">y</span><span class="o">][</span><span class="n">x</span><span class="o">]</span><span class="p">:</span> <span class="n">dec</span> <span class="n">n</span>
<span class="n">univNew</span><span class="o">[</span><span class="n">y</span><span class="o">][</span><span class="n">x</span><span class="o">]</span> <span class="o">=</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">3</span> <span class="ow">or</span> <span class="p">(</span><span class="n">n</span> <span class="o">==</span> <span class="mi">2</span> <span class="ow">and</span> <span class="n">univ</span><span class="o">[</span><span class="n">y</span><span class="o">][</span><span class="n">x</span><span class="o">]</span><span class="p">)</span>
<span class="n">swap</span> <span class="n">univ</span><span class="p">,</span> <span class="n">univNew</span>
<span class="n">sleep</span> <span class="mi">200</span></code></pre></figure>
<p>Nim is a rather new language, and it took me some time to choose it. What I love about the language:</p>
<ul>
<li>Python-like syntax</li>
<li>Statically typed, compiled</li>
<li>High performance (same ballpark as C/C++)</li>
<li>Garbage Collector that can be controlled for soft real-time</li>
<li>Produces executables without dependencies (compiles to C)</li>
<li>Easily interfaces C libraries (compiles to C)</li>
<li>Clean and powerful metaprogramming</li>
</ul>
<p>On this blog I will report about my progress in the development of HookRace and my current Nim adventures.</p>