Work Posts · HookRace Blog

Workload Capture & Replay

2026-02-10T00:00:00-05:00

When customers hit issues in production, it can be an effort to locally reproduce them, especially when external sources are involved. Reproducing issues is useful not just to figure out the root cause, but also to verify the fix and add a regression test. The newly introduced workload capture & replay tooling records a Materialize instance’s state as well as recent queries and ingestion rates, then replays them in a Docker Compose environment with synthetic data. In this blog post I’ll show how it works and talk about some of the challenges and future work.

Read the rest of the blog post over on the Materialize blog.

Speeding up Materialize CI

2025-08-08T00:00:00-04:00

In the previous post I talked about how we test Materialize. This time I’ll describe how I significantly sped up our Continuous Integration (CI) Test pipeline in July, especially for pull requests that require a build and full test run. The goal is to make developers more productive by reducing the time waiting for CI to complete.

We always kept CI runtime in mind, but it still slowly crept up over the years through adding tests, the code itself growing larger, as well as hundreds of minor cuts adding up.

Read the rest of the blog post over on the Materialize blog.

How to Use the Materialize Emulator

2024-10-16T00:00:00-04:00

In our last blog about our Quality Assurance (QA) team, we gave an overview of the QA process, including our software and testing methods. One of our key tools during testing is the Materialize Emulator, a Docker image that allows you to maintain a locally hosted version of Materialize.

Read the rest of the blog post over on the Materialize blog.

The CI Flake

2024-08-16T00:00:00-04:00

I analyzed a flaky test failure in our Materialize CI today:

$ docker compose up -d --scale default=0 default
no such service: default
mzcompose: error: running docker compose failed (exit status 1)

I had seen this error already once or twice in the last year, but it was incredibly rare in our Continuous Integration (CI) runs, and never happened locally. As usual, there were more pressing product issues to debug, so I never looked into it. But last week I switched most of our CI tests to run on Hetzner Cloud instead of AWS to save some money. Suddenly this issue started occurring more often in CI, so my thinking was that it must somehow be timing-dependent.

Before investigating my first instinct was that we are not writing the docker-compose.yaml file properly, which had already led to flaky calls to Docker Compose before (source code):

file = self.files.get(thread_id)
if not file:
    file = TemporaryFile(mode="w")
    os.set_inheritable(file.fileno(), True)
    yaml.dump(self.compose, file)
    self.files[thread_id] = file

So yesterday I added a file.flush() after the yaml.dump and hoped to be done with it. This morning I woke up and the issue was still occurring!

Based on the logged docker call this should be the code causing the problem (source code):

def handle_composition(
        self, args: argparse.Namespace, composition: Composition
    ) -> None:
        if args.workflow not in composition.workflows:
            # Restart any dependencies whose definitions have changed.
            # This is Docker Compose's default behavior for `up`, but
            # not for `run`, which is a constant irritation that we
            # paper over here. The trick, taken from Buildkite's
            # Docker Compose plugin, is to run an `up` command that
            # requests zero instances of the requested service.
            if args.workflow:
                composition.invoke(
                    "up",
                    "-d",
                    "--scale",
                    f"{args.workflow}=0",
                    args.workflow,
                )
            super().handle_composition(args, composition)
        else:
            [...]

Running the test in an endless loop locally had no success of reproducing the issue:

while true; do
    bin/mzcompose --find skip-version-upgrade down
    bin/mzcompose --find skip-version-upgrade run default || break
done

I modified the conditional in the above Python code to be if True: and surely that reproduces the issue:

$ bin/mzcompose --find skip-version-upgrade run default
$ docker compose up -d --scale default=0 default
no such service: default
mzcompose: error: running docker compose failed (exit status 1)

So how is it possible to be landing in this code path sometimes? I checked how the composition.workflows are generated and it looks quite complex indeed (source code):

mzcompose_py = self.path / "mzcompose.py"
if mzcompose_py.exists():
    spec = importlib.util.spec_from_file_location("mzcompose", mzcompose_py)
    assert spec
    module = importlib.util.module_from_spec(spec)
    assert isinstance(spec.loader, importlib.abc.Loader)
    loader.composition_path = self.path
    spec.loader.exec_module(module)
    loader.composition_path = None
    self.description = inspect.getdoc(module)
    for name, fn in getmembers(module, isfunction):
        if name.startswith("workflow_"):
            # The name of the workflow is the name of the function
            # with the "workflow_" prefix stripped and any underscores
            # replaced with dashes.
            name = name[len("workflow_") :].replace("_", "-")
            self.workflows[name] = fn

Since we are reading and analyzing the mzcompose.py file, something might be running in parallel and corrupting/overwriting it. If I empty the file I can indeed reproduce the issue with the original code:

$ echo > test/skip-version-upgrade/mzcompose.py
$ bin/mzcompose --find skip-version-upgrade run default
$ docker compose up -d --scale default=0 default
no such service: default
mzcompose: error: running docker compose failed (exit status 1)

It feels like I’m getting closer, but nothing should be running in parallel, even in CI each agent is running in isolation, the git pull is long finished at this point based on the CI logs, so it should not interfere. As an additional frustration, I’ve been wondering why this flake is mostly (but not only) occurring when running this test? The mzcompose.py file in question is one of the simplest ones around.

Until here it felt like I was getting closer, but now I had no more leads. I went as far as searching for Python bugs around inheritance that could explain this issue, but if Python was buggy in such basic behavior, this code definitely wouldn’t be the first one to run into it. Could it be a cosmic ray?!

Since this issue seemed unexplainable without the universe conspiring against me, I went back to the basics and checked what else is running in CI and I found something extremely suspicious just before the actual test execution in our CI script (source code):

# Start dependencies under a different heading so that the main
# heading is less noisy. But not if the service is actually a
# workflow, in which case it will do its own dependency management.
if ! mzcompose --mz-quiet list-workflows | grep -q "$service"; then
    ci_collapsed_heading ":docker: Starting dependencies"
    mzcompose up -d --scale "$service=0" "$service"
fi

Oh no! There is another $service=0 call here, my entire investigation so far might have been useless! From looking at the CI output it’s not clear which one is actually running, but the fact that I’ve never seen the flake locally points to the new suspect. After all this code is only running in CI, and not locally, all in the name of cleaner CI logs! Checking the CI output there is a strange line occurring:

write /dev/stdout: broken pipe

Well, there is a pipe between the mzcompose and grep processes, but why would it break? Let’s run the mzcompose command in isolation:

$ bin/mzcompose --find skip-version-upgrade list-workflows
default
test-version-skips

It lists all workflows, of which we have two. The grep command is only looking for the default line. Apparently, grep is smart enough to exit immediately when it finds a match when used with -q (manpage):

-q, --quiet, --silent
       Quiet; do not write anything to standard output. Exit
       immediately with zero status if any match is found, even if
       an error was detected. Also see the -s or --no-messages option.

Easy enough to verify, I ran the true command, which exits instantly:

$ bin/mzcompose --find skip-version-upgrade list-workflows | true
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

So this broken pipe output is indeed explained by grep exitting early. But why does the mzcompose process breaking lead to the code in the if’s body being executed? Normally the first command in a pipe failing would be ignored, since the second command succeeded, and I’m seeing the return code being 0 here. Checking the beginning of the script reveals that we use the pipefail setting:

set -euo pipefail

Bash’s manpage explains:

The return status of a pipeline is the exit status of the last command, unless the pipefail option is enabled. If pipefail is enabled, the pipeline’s return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully.

Typically, Python buffers its output when it doesn’t write directly to a terminal, so I still can’t reproduce the issue with grep locally. Then I remembered that we had an issue with our Python tests’ stdout and stderr output being garbled, which made us enable PYTHONUNBUFFERED in our ci-builder Dockerfile:

# Ensure that all python output is unbuffered, otherwise it is not
# logged properly in Buildkite
ENV PYTHONUNBUFFERED=1

With this revelation the flake was easy enough to reproduce. Just running this would fail occasionally, still less than 1% on my system:

$ set pipefail
$ PYTHONUNBUFFERED=1 bin/mzcompose --find skip-version-upgrade list-workflows | grep -q "default"
BrokenPipeError: [Errno 32] Broken pipe

Adding a small sleep between each print leads to a reliable failure:

diff --git a/misc/python/materialize/cli/mzcompose.py b/misc/python/materialize/cli/mzcompose.py
index 3a2f5dbb81..725150ed00 100644
--- a/misc/python/materialize/cli/mzcompose.py
+++ b/misc/python/materialize/cli/mzcompose.py
@@ -319,6 +319,7 @@ class ListWorkflowsCommand(Command):
         composition = load_composition(args)
         for name in sorted(composition.workflows):
             print(name)
+            time.sleep(0.1)


 class DescribeCommand(Command):

Without PYTHONUNBUFFERED it would have taken us 8 KiB of output before we’d run into the broken pipe, none of our tests has that many workflows.

That concludes this long investigation into what turned out to be an interesting flake that we’ve been seeing in CI! The trivial fix is removing the -q from the grep call, but we could also have disabled pipefail for that one pipe or we could have unset PYTHONUNBUFFERED. One lesson from this debugging session is that even simple behaviors can combine in unexpected ways to create strange flaky failures. In this case all of these conspired to make this CI flake possible:

Misdirection: Run different code in CI compared to locally
Switch to another kind of server with slightly different performance characteristics
Disable Python’s output buffering
Enable Bash’s pipefail
Make grep run quietly

In hindsight I realized why this specific skip-version-upgrade/mzcompose.py was having the flake most often. It is one of the smallest test files:

$ wc -l test/*/mzcompose.py | sort -n
      39 test/pg-rtr/mzcompose.py
      43 test/mysql-rtr/mzcompose.py
      67 test/debezium/mzcompose.py
[...]
      85 test/skip-version-upgrade/mzcompose.py
[...]
     966 test/0dt/mzcompose.py
    1333 test/bounded-memory/mzcompose.py
    1916 test/limits/mzcompose.py
    4767 test/cluster/mzcompose.py

But other than the test definitions that are even smaller and never had this no such service: default error, the skip-version-upgrade/mzcompose.py file contains multiple workflows, so the smaller files could not run into the issue at all:

$ grep "def workflow_" test/*/mzcompose.py | cut -d":" -f1 | \
  uniq -c | sort -n | grep -v "   1 "
   2 test/skip-version-upgrade/mzcompose.py
   3 test/canary-environment/mzcompose.py
[...]
  57 test/cluster/mzcompose.py

Testing Materialize

2024-05-14T00:00:00-04:00

I joined Materialize’s Quality Assurance (QA) development team over a year ago. Since our team is small, we have to be conscious about focusing our time on the most impactful testing and test tooling.

Our goal is to find issues in Materialize as early and as efficiently as possible, ideally before a pull request even lands. Despite our small team, Materialize has a huge surface area.

Read the rest of the blog post over on the Materialize blog.

Database Testing at Yugabyte

2023-01-14T00:00:00-05:00

YugabyteDB is a cloud-native database for business-critical enterprise applications. It is designed to provide continuous availability as well as horizontal scalability, while retaining a strong set of RDBMS features. This objective creates a strong quality incentive for us in the Yugabyte Quality Assurance (QA) team. As a member of this team, I am giving an overview of the testing philosophy, approaches, and implementations for YugabyteDB.

Read the rest of the blog post over on the YugabyteDB blog.

€9 Ticket

2022-08-20T00:00:00-04:00

As a relief for rising energy costs and inflation Germany is offering nearly-free public transportation for the months of June, July and August. For just €9 per month you can take any kind of regional public transportation in all of Germany. That’s basically everything except the IC/ICE for long distance. This means that a monthly ticket is now as cheap as what I would normally pay for a single route. In this post I want to talk about my experiences with this €9 ticket.

Update 2022-09-01: The €9 ticket ended yesterday without a similar replacement system in place. So many people are again back to paying about €100 per month for much smaller regional tickets instead (see for example this German Reddit thread. Financially it makes more sense for me personally to go back to driving. (I hope the gasoline in the car is still good after three months.) My hope is that these three months will stay in people’s minds in Germany and shape the future of public transportation to be cheaper and simpler:

Image via Reddit

Train Stations

In the above graphic you can see the train stations in our area. Since we are situated in the Rhine-Neckar metropolitan region the railway network is pretty good here.

(map made with umap, routes made with GraphHopper)

The closest train stop from us is St. Ilgen-Sandhausen at 2 km distance. It offers direct connections to the cities of Heidelberg (160k inhabitants), Mannheim (309k), Karlsruhe (313k), Darmstadt (159k) and Frankfurt (753k). For other directions I prefer going to a different train station since switching times are either too long or there is too much risk of missing the connecting train. Only 89% of regional trains are on time in Germany, with some routes and stations having much worse rates. This is a far cry from Switzerland’s 95% punctuality with an even stricter definition of 3 minutes versus 5 minutes delay in Germany.

Another closeby train station we commonly use is HD-Weststadt/Südstadt at 6 km distance, which is good to cycle and can also be reached via Tram. Occasionally I used Heidelberg Hauptbahnhof as well as Wiesloch-Walldorf, both at slightly under 10 km distance, which are a bit larger and thus are used as stops for Regional-Express trains. These go faster than the Regionalbahn and S-Bahn trains and thus also have fewer stops.

Cycling Trips

The above map contains some of the cycling trips I took by combining them with the €9 ticket. (map made with umap, routes made with GraphHopper)

Last time I posted about cycling and work was about commuting by bicycle for a year. I kept this up and was super happy with this way of getting to SAP’s headquarter in Walldorf and the included exercise. After a while I moved together with my wife and my bike commute lengthened to 10 km each direction, which was still perfectly fine. Then Covid forced me in March of 2020 to work from home I switched to regular cycling trips in the afternoon since I started working early in the morning, in order to have meetings with colleagues in Korea and China. Two 50 km cycling trips per week got me to the same level as the daily commute used to do, and I have a nice route I regularly take, which covers a lot of forest and lake.

In January I switched to a new job at Yugabyte. As the database we develop is distributed, so are the people I work with. Since most meetings fall into US times, I am now working more during the afternoon and evening. This means my cycling trips moved to the morning instead.

Previously I had a job ticket for three years, but other than the €9 ticket it limits the range I can take public transportation in, so that many of my destinations would be barely outside of the covered range. When looking at a map of Germany with all the different transport associations, each with their own tickets, prices and rules, I feel reminded of historical maps of the Holy Roman Empire:

Good luck figuring out what ticket to get to combine with your job ticket. Luckily the €9 ticket changed this, at least temporarily, making life simpler.

I used the ticket during the working days to discover new cycling routes. By cycling one direction and taking a direct train connection back, I can cover nearly twice the distance as usual with the same starting and stopping point. With this method I took many routes I had never taken before, discovering beautiful cycling paths along the Neckar river for example. Luckily the trains are not too crowded during working days, especially outside of rush hour. The cycling usually takes around three times as long as the train ride back.

The image above shows an almost empty train during a working day.

Even with a single train connection things can go wrong though. One time the train in front of ours started burning and so we got delayed for an unknown time. But since I had my bicycle with me, that meant I could just cycle back the rest of the way and get some extra exercise for free. If I was really out of energy I could have waited in the train, but often busses are used as a replacement, and bicycles on busses don’t fit well.

The weather was unusually dry the last months, the last few days had the first few drops of rain I remember since the beginning of June. This is of course bad for nature here, as I saw at the Rhine river, which is much lower than usual. For cycling it’s quite good though, only got wet once.

Jogging

This amount of cycling of course meant that my bicycle had to get some repairs done. While I had it at the repair shop, I tried out jogging from Heidelberg back to my home town Leimen. The route goes through the “Kleiner Odenwald” forest for nearly the entire way, and the rest are some fields. It goes through Heidelberg’s highest mountain, the Königstuhl, with a nice view of the city and surrounding area.

The “Ladder to Heaven” in the beginning with its 1300 steps accounts for most of the elevation and is still too tough for me to do at faster than walking speed. After I got my bicycle back I still kept doing this jogging route a few times per week.

Nearby Excursions

There are lots of cities and nice nature nearby. So we used public transportation for excursions on every weekend. It was noticable that the trains were crowded, and at times people couldn’t even enter the train anymore. When going to the cinema we noticed that we should really catch the last train at 22:30, since the next connection would take all night.

One time the train back from Frankfurt was late, then got cancelled. The next train was also delayed, and now overcrowded because it had twice the passengers. Unfortunately it took me too long to realize that with more than 20 minutes delay, we should be able to take high speed ICE trains as a replacement and get the cost reimbursed by Deutsche Bahn.

You still need to wear masks as a Covid prevention on public transportation in Germany. This is basically the only remaining occasion, other than at the Doctor’s office, where you have to wear masks. Even with all the travel using public transportation and basically no distancing we haven’t caught Covid as far as we are aware based on symptoms and some rapid tests.

Eating Out

My wife picked out many great restaurants in nearby cities for us to try out. By car it would not make sense to just drive to another city for a dinner, but using public transportation we had no problem spending an hour on a train, where you can comfortable talk or surf the web, to reach our destinations. This probably has a really good economic effect on restaurants and cafes, many were so fully packed that we got the last seats or had to reserve beforehand.

Family Visits

For trips to cities and cycling trips we mostly take direct train connections. That usually works quite well. To visit family however the destinations are smaller villages, so the connections are worse and there is no direct connection at all. So a 40 minute drive turns into a 2 hour train ride with 3 different trains instead. Coincidentally this is the same time I need for the route by bicycle.

There is a relatively high chance of missing a connecting train, or of getting stuck in the middle of the route. One time a truck crashed into the train bridge, so our train couldn’t continue. Instead of waiting around for an hour or two we took the train back home and, for the only time during these three months so far, had to switch to the car to keep our timeline. At least the highway was quite empty, there seems to be some conflicting data on whether the €9 ticket actually has an effect on reducing car traffic though.

Getting Visits

While my wife and I didn’t do any long distance travel using the €9 ticket, others did. I had a co-developer of DDraceNetwork visit all the way from Berlin using the €9 ticket. For people who have time it has been much easier to visit others with the €9 ticket, without having to worry about cost.

While on Vacation

While we didn’t use the €9 ticket to go to a vacation, it was still useful during a car-based vacation. While driving to Berchtesgadener Land we saw a train line that seemed to lead to Berchtesgaden. So we stopped at the closeby train station and spontaneously took the train instead, which lead us through beautiful sceneries.

On the way back from the same vacation we had a stop in Munich. Instead of driving the car all the way into the city center I checked for the closest free Park and Ride garage from the highway. After parking the car there, the train was unfortunately cancelled, so we had to take a bus and metro, but still reached the city center quickly.

After having used public transportation a lot back at home, we also felt more comfortable using it on vacation in other countries, even if it costs a bit more there. So on Mallorca we explored the entire island using the extensive bus network instead of renting a car. The photo above shows the most distant bus stop we reached, at a lighthouse at the end of a long curvy road. Next week we are taking the TGV to Paris, which tops out at 320 km/h and manages the 550 km route in just 2:36, an average of 212 km/h. The image below is luckily not the TGV:

Going to the airport by train worked fine, but we had to get there a few hours earlier in case was some train delay causing us to miss the connecting train. On the way back there indeed was a huge delay, with a train line being closed down for hours. After a slow trickle of information from the train operator, we had to figure out an alternative route on our own, since the DB app was still showing routes that could not be operated for the next few hours because of the train line closure.

Conclusion

There are still two weeks left for the €9 ticket and I’m planning to make the best use of them. So far I’ve been taking a trip on most days, so it’s definitely been worth it and will be missed.

Unfortunately the €9 ticket won’t be prolonged. The high public interest has lead to some plans for a relatively cheap follow-up ticket, but again with some regional limitations for the cheap ticket. Other countries are even implementing entirely free public transport meanwhile, so there is some hope for the future.

One year of cycling to work

2018-02-20T00:00:00-05:00

Exactly on this day, one year ago, I came back from a one month long trip to Taiwan, went straight to work from the airport and immediately moved into a new apartment after work. Since then I have cycled to work nearly every day.

Fullscreen (map made with umap, routes made with GraphHopper)

Alternatives and motivation

Previously I lived in Heidelberg, 15 km from work and usually took the train, only cycling to work about once a week. But my new apartment was significantly closer to work, only 6 km, which takes just 15-20 minutes to cycle, so it became much easier to use the bicycle to get to work every day.

Taking public transportation instead is possible, and takes about 35-40 minutes. I only do that when I want to go to the city straight after work instead of going home first. For me personally public transportation is more useful in combination with the bicycle on longer trips.

Sometimes I take public transportation to work and just walk back in the evening, which can be great for listening to audiobooks.

For about a month I had borrowed a car and sometimes used it to go to work. Without traffic the driving would take 12 minutes, but with traffic it’s usually 15-25 minutes. Additionally I have to get a parking spot and walk further to the office, so using a car doesn’t save time compared to cycling.

Advantages

I remember feeling tired when getting to work early in the morning using public transportation. After all you just keep sitting and standing, never really jump starting the body. While cycling to work I certainly wake up and feel more energetic and focussed when I arrive.

Having a clear separation of work and private life is important for me, both by using different spaces for each as well as having a strict time separation. So after I get out of work and get onto my bicycle I will still be having work-related thoughts circling in my head. But while cycling I might take a quick stop to write down any insights that I have or tasks that I forgot to finish. Then I can finally shut down and clear my brain from work by enjoying the ride, the views and the physical exercise.

I find the simplicity of cycling for locomotion beautiful. I never have to be stuck in a traffic jam, feeling stressed by moving at a snail’s pace. Even if cycling is slower the feeling of making progress is stronger. I never have to be waiting for a late train, a bus that left too early or be stuck inside of a train while there are problems with the power line. In the worst case I can pick up my bicycle and walk around some obstacle or fix a flat tire in a few minutes. Being outside in nature, at least for half an hour every day, feels good.

Combined with the gym at work I feel like I’m getting enough sports to be reasonably fit without having to spend much time on it. I regularly get home from work at 17:00 with all work and sports done and can enjoy the rest of the day without any worry.

When I’m using the bicycle to get to work already, I feel like the barrier to combining this with larger trips is lowered. I occasionally do 40-50 km cycling tours before or after work, depending on the weather and my mood.

Accommodation

At home my bicycle is locked into a room in the basement, making it reasonably secure and still easily accessible. At work we have roofed-over parking spaces for the bicycles with robust metal bars for locking. Some of the cycle racks even have two floors with an easy mechanism for getting your bicycle down from there. When I don’t need my lock I will just leave it there after work for the next day.

Multiple buildings at work have special showers for cyclists. But to be honest I don’t use them after my usual route to work, for three reasons:

It is cooler in the morning.
The distance is short enough.
I cycle slower when going to work in the morning than when going back.

Especially in the summer it is nice to do a two hour cycling trip at 5 am and go straight to work afterwards. Then the showers are a godsend of course.

The journey is the reward

Close to my home I have to cross a red light, which is also where I had the only accident in this year. A car driver ignored my right of way and so I had to break hard and ended up sitting on their hood. But luckily that didn’t result in any injuries or damages. (Edit: I have never had an accident involving a car before, so this is probably an outlier.)

After exiting the industrial area I am riding exclusively on small field roads. Especially in the morning this makes for nice sunrises with fog on parts of the fields. Along the way there is also a fishing lake.

The way tunnels below train tracks, runs over a stream, tunnels below a large street and finally bridges over another large street. This means there are no large intersections where you have to wait or be cautious. In consequence the ride is quite relaxing.

The only potential stop along the way is a small airfield where planes are occasionally landing and starting. But that’s fun to watch, so I don’t mind waiting for a minute or two.

Numbers

There are about 250 work days in a year in Baden-Württemberg, excluding weekends and public holidays. Of those I was missing for 34 days for vacations, being sick and doing home office. On 11 days I used public transportation, on 5 days a car. That leaves 200 days on which I cycled to work.

The distance to work is 6 km each way, so in total I cycled 2400 km just to get to work and back. I need 20 minutes to work and 15 minutes back, so 117 hours were spent cycling.

I had two flat tires, one accident and needed one major part replacement in the entire year. But to be fair I also cycled in my free time and it was the first large replacement my bicycle needed since I started using it 15.000 km ago.

Animal nature

On one day in summer I counted the wild animals that I saw on my short trip to work, which included 4 rabbits, 1 deer with fawn, 1 white stork with young in the nest as well as a group of 21 white storks in the fields. A few weeks ago I was happy to see an elusive black stork flying towards the closeby forest for the first time. On another occurence there was a Heron successfully catching a snake.

Apart from wild animals there are of course many people walking their dogs, and the pet home on the way features lots of dogs behind the fence as well as a few cats walking around outside.

Winter is coming

Appropriate clothing is important since cycling exposes you to wind and precipitation.

In summer a t-shirt and short pants are usually fine. For colder times dual-function pants for office and cycling are great, otherwise jeans also work.

During rains I certainly don’t want to wear jeans. Then I also need to bring another set of clothes to switch to once I’m in the office. The most important measure for rainy days is to have good timing, since I have flexible work times and the rains are usually not equally strong for hours. Weather forecasts can be remarkably inaccurate though.

I use wind-proof jackets and shoes as well as gloves and a balaclava in winter. Snow and ice are rather rare and not so strong here, so they are not a big issue. The above photo was taken two years ago and that represents about the worst winter conditions I ever had to cycle in.

Conclusion

I’ll keep cycling to work in the foreseeable future. I don’t think this post is of much use to anyone, but I had fun reflecting my last year of cycling to work. Thanks for reading. Comments on Hacker News.

Turning Down a Blockchain Job Offer

2018-01-22T00:00:00-05:00

I have recently received a job offer to work on a blockchain implementation. While the offer was very generous, I had to turn it down. In this post I want to collect the thoughts that went into my decision process leading to this conclusion.

Introduction

I have not touched blockchain technology much before, other than reading the original Bitcoin paper shortly after it was released and accepting Bitcoin donations for DDNet. But I have always used those donations pretty much immediately, and thus I have no investment in cryptocurrencies either way and am thus impartial to their fate.

The Good

I consider the Bitcoin idea to be quite interesting as an electronic cash system that doesn’t require any trusted intermediary. This is a small and simple idea that combines old cryptographic ideas into something new.

The job I got offered had a significantly higher salary attached than my current job. The new job would have been remote, thus enabling me to spend more time with my girlfriend and live in any location I wish.

I would have been able to work in a nice programming language, developing a blockchain implementation from the ground up with a small group of colleagues.

The Bad

Proof of work in the form of mining is currently the most common form of trustless consensus in blockchain systems. It requires immense amounts of computational hardware in the form of ASICs and GPUs as well as energy to supply them. The only function of this is to prevent any one actor to take over more than 50% of the mining market and thus controlling the currency. This is a large price to pay for replacing trust.

Looking at it the other way: Trust can be seen as a shortcut that humanity has used for a long time in order not to require expensive proof of work. For most applications this still works fine.

Companies are doing initial coin offerings (ICOs), giving out their own cryptotoken to public investors, circumventing a proper IPO process. While companies often claim that their tokens will have another purpose, this is usually not true.

Reading about blockchain is horrible because of all the hype, cargo cult and politics surrounding it. Everyone who is talking positively about cryptocurrencies and making hyperbolic claims seems to have invested in them. So they might be trying to pump up the value of cryptocurrencies in order to profit themselves. It might be that this aspect is so dominant that the actual technological details are less relevant. It feels like talking to a religious group that is trying to convert you to the one true faith that will solve all problems you and society have. I wish it was possible to filter out everyone who had money invested in the fate of cryptocurrencies in those discussions.

It seems like people already know that they want to use a blockchain even before they understand what the problem is, basically a solution in search of problems. The main use of cryptocurrencies right now seems to be as a speculation device.

The Conclusion

In the end I guess I don’t want to be associated with the current state of the blockchain ecosystem. So even if the job offer is great, I have to deny it. It took me a long time to realize this for myself and getting my priorities straight.

Discussion on Hacker News.

HANA C++ Development Environment and Processes

2017-12-08T00:00:00-05:00

I started working as a C++ developer in the HANA Core Platform team at SAP in Walldorf, Germany more than a year ago. In this time I have gotten some insights into the development environment and processes. I will use this post to illustrate them by the example of adding a small new feature and explaining the steps on the way of getting it released. Some of this will be specific to HANA, some to my team inside HANA, some to my own system.

Compilation

HANA is the in-house database powering many of SAP’s products. It is written in C++ with Python tests, and the entire code base lives inside of a single git repository. Hundreds of developers from all around the world are developing on about 10 million lines of C++ code and 15 million lines of Python tests.

Since HANA is deployed on Linux exclusively, many developers are using Linux on their workstations as well. So far Windows is still supported as a development environment, but this will change in 2019, leaving Linux as the only choice. During day-to-day work you still get to interact with Windows a bit (more than I would prefer), since the Laptop has Windows with the traditional Microsoft Office products on it. But since the actual development happens on Linux, I am happy enough, being able to use the xmonad window manager and software environment I have gotten used to over the last decade.

HANA is deployed on some rather impressive machines, so in order to test the code adequately the developer workstations are quite beefy as well. I am typing this text from a 20 core / 40 thread Xeon E5-2660 v3 CPU with 128 GB of RAM.

Still compiling HANA is no quick feat. Expect something around 2 hours to build it from scratch on your local machine (illustrated below), and be prepared for some heat output, which is especially noticeable in summer. About half of the colleagues have chosen to move their workstations to the data center and access it remotely instead of working on it locally. Since they are still in one of our SAP buildings in Walldorf, the latency is low enough that a direct X connection is fast enough to be nearly undistinguishable from a local application.

So if even your beefy developer machine is not enough, how do you compile the product faster? Obviously by combining all of the developer machines together in a compile cluster:

Using the compile cluster the build time gets reduced to 16 minutes and now linking takes half of that time. Right now we are still using the gold linker, but after bug fixes lld should cut down the linking time significantly.

Coding

But for the code in my team we have another approach. Instead of running the full database and testing using Python tests we have extensive unit tests, literally covering our entire code. Every single line is covered and about 98% of all regions. In order to submit any new code you have to write a unit test that will cover this exact code and include it in the same change.

Expect 5 minutes to build and run the unit tests from scratch, while building and running them after a change can be as fast as 15 seconds if you didn’t change much and thus not much has to be recompiled.

People use quite a variety of IDEs and code editors, with Sublime Text, Qt Creator, Eclipse, Visual Studio, emacs and vim being popular choices (in no particular order). Personally I like to work directly on terminals, so I use vim with the YouCompleteMe semantic code completion (and go to definition, type analysis and some more) based on clang/llvm. I use a custom YCM version which also shows me the size and alignment of a variable along with its type:

plangen::PipelineBuilder & => hex::plangen::PipelineBuilder & (size: 128 B, align: 16 B)

Using vim and other command line tools gives me the advantage that I can work just fine as long as I have an SSH connection to my workstation, nothing else required.

Let’s start by coming up with a useful new feature, tracing the name of the DummyNoRun operator:

#pragma once

#include "hex/planex/framework/OperatorBases.hpp"
#include "hex/planex/planviz/HasReversedConnection.hpp"

#include "warnings/clang_warnings_hard.h"
#include "warnings/gcc_warnings_veryhard.h"

namespace hex {
namespace operators {

class DummyNoRunOp final
    : public planviz::HasReversedConnection,
      public planex::NoRunOperator
{
    void traceOperatorName(ltt::ostream& os) const;
};

} // namespace operators
} // namespace hex

#include "warnings/clang_warnings_end.h"
#include "warnings/gcc_warnings_end.h"

The implementation is quite simple as well:

void DummyNoRunOp::traceOperatorName(ltt::ostream& os) const
{
    os << "DummyNoRunOp";
}

For HANA development we use C++14 at this moment. The only unusual thing is that we don’t use the STL, but instead have our own implementation of it, the LTT, which enforces allocator use.

Our build system wraps around cmake, so it is quite reasonable to use. After adding the cpp file to CMakeLists.txt we can locally run our unit tests and verify that nothing broke so far.

But when we’re compiling the code with hm b -b Optimized unitHexOperators (read as: HappyMake build Optimized build [with optimizations and assertions] of the unitHexOperators target) we get an unexpected result:

In file included from ../../hex/plangen/PredInitOpGenerator.cpp:14:0:
../../hex/operators/table/DummyNoRunOp.hpp:16:10: error: ‘virtual void hex::operators::DummyNoRunOp::traceOperatorName(ltt::ostream&) const’ can be marked override [-Werror=suggest-override]
     void traceOperatorName(ltt::ostream& os) const;
          ^~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
ERROR: subcommand failed

Our warning levels are quite high and all warnings are converted to errors. We even enable warnings in headers (and disable them again at the end in order to not affect other headers that might be included later). In this case the fix is quite simple:

    void traceOperatorName(ltt::ostream& os) const override;

Now our code builds and the unit tests succeed:

$ hm b -b Optimized runtests_hex
...
[==========] 728 tests from 147 test cases ran. (11617 ms total)
[  PASSED  ] 728 tests.

  YOU HAVE 2 DISABLED TESTS



Build started:	2017-11-29 16:26:40.459482
Build finished:	2017-11-29 16:26:52.988541
Elapsed time:	12.529s
Command count:	11 (0.9 per sec)
Log directory:	/home/dXXXXXX/src2/build/Optimized/hm_log/build/24
SUCCESS

We can also run the code coverage based on our unit tests locally using CheckBot. Unsurprisingly we will find out that our code is not covered yet:

###############################################################
                  CheckBot - Review Details
###############################################################

###############################################################
                      CheckBot - Summary
###############################################################

______________
General Issues
______________
[CheckHexCodeCoverageClang] (error) hex/operators/table/DummyNoRunOp.cpp: 3 of 3 lines uncovered, 0.00% coverage!
[CheckHexCodeCoverageClang] (error) Line coverage only at 99.99%
[CheckHexCodeCoverageClang] (info) Coverage report can be found at 
file://///home/dXXXXXX/src2/build/linuxx86_64-clangcov-release_hex_with_code_coverage/hexCoverageClang/index.html

CheckBot detected 3 issues (2 errors, 1 info).

	error ... CheckHexCodeCoverageClang  (681748.81 msec)

Overall score: -2

(Legend: ok = +1, info = +1, warning = -1, error = -2)

At the bottom of the report we see the “Overall score: -2”, which indicates that our change would not be allowed to be merged by Gerrit. So let’s take a closer look, inside of the linked report we find our code:

Google Test/Mock based unit tests are used to test every part of the code in our team. 100% unit test line coverage is achieved with a development version of llvm-cov. Compared to the gcc based gcov/lcov coverage tools, the LLVM based llvm-cov is in active development, runs much faster and can show you what region of a line is actually covered, allowing finer granularity.

So let’s write a unit test to satisfy the 100% line coverage requirement:

#include "hex/operators/table/DummyNoRunOp.hpp"
#include "hex/test/HexTestBase.hpp"
#include "hex/planex/test/sandbox/SimpleOperatorSandbox.hpp"

#include "warnings/clang_warnings_hard.h"
#include "warnings/gcc_warnings_hard.h"
#include "warnings/msvc_warnings_hard.h"

using namespace ::testing;

namespace hex {
namespace operators {
namespace test {

class UnitDummyNoRunOp : public HexTestBase<>
{
protected:
    planex::SimpleOperatorSandbox<DummyNoRunOp> m_sb{alloc()};

    ltt::ostringstream m_oss{alloc()};
};

TEST_F(UnitDummyNoRunOp, traceOperatorName)
{
    m_oss << planex::Operator::TraceName{m_sb.op()};
    EXPECT_THAT(m_oss.c_str(), StrEq("UnitDummyNoRunOp"));
}

} // namespace test
} // namespace operators
} // namespace hex

For more coarse-grained testing there are also JSON based component tests as well Python based integration tests available.

Local Verification

We should also make use of some static code checkers, like clang-format and clang-tidy, before we push the change for remote testing. After all, this ensures a consistent style, fixes some common problems and reduces the mental load on the manual reviewers, since they can rely on the tools instead of complaining about the same trivial nitpick on every change:

###############################################################
                  CheckBot - Review Details
###############################################################

_____________________________________________
hex/operators/table/test/UnitDummyNoRunOp.cpp
_____________________________________________

[CheckClangFormat] (info) This file is subject to a clang-format code style. CheckBot can reformat the file, see [...]

line 2,0 - 2,35:
#include "hex/test/HexTestBase.hpp"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[CheckClangFormat] (warning) 

 #include "hex/test/HexTestBase.hpp"

should be deleted

line 3,0 - 3,0:
#include "hex/planex/test/sandbox/SimpleOperatorSandbox.hpp"
^
[CheckClangFormat] (warning) 

 #include "hex/test/HexTestBase.hpp"

should be added

Luckily clang-format can even fix the problem on its own by just adding a parameter.

Remote Verification

Finally we can push the change to Gerrit. Now our local work is done and we can start working on something else. Meanwhile automated builds and tests will happen on Linux and Windows machines, as well as a coverage run and various static code analyzers (that we were too lazy to run locally).

By default just our unit tests and component tests are compiled and executed. But if you want to build the full HANA database and run the Python tests a separate profile is available for that and can be turned on in the Gerrit web interface for specific changes.

Code Review

Once we have verified that everything works fine on Linux with GCC and Clang as well as Windows with MSVC, we can add a reviewer to our change and set the change to “Ready for Review”. A second reviewer will be chosen automatically. If you are relatively sure that your change will survive the tests and don’t feel like waiting for them, you can of course also add the reviewers straight after submitting the change. But your reviewers might not be happy if they start reviewing already and meanwhile you are pushing new patch sets to fix compilation, tests as well as automatic warnings and suggestions.

For non-trivial changes further patch sets would follow to integrate the suggestions by the reviewers, until everyone is (reasonably) happy. Finally the change can be submitted and is out of our mind.

Merging

Except that there is still one thing missing. Our change is now inside of our component’s hex branch, but not yet in the global orange branch. The merge from hex to orange will run a huge number of correctness and performance tests before allowing our changes in. That’s why we only run it after we have collected a few changes. We also use a staging branch hex2orange.

Initially we merge the hex branch into hex2orange and try to merge hex2orange into orange.
Meanwhile everyone can keep developing new features as well as bug fixes on the hex branch.
If the merge from hex2orange to orange fails, a fix for the issue will be submitted to hex and can finally be cherry-picked to hex2orange.

But we don’t merge hex into hex2orange again, until we landed in the orange branch. Otherwise the new features from hex could cause new test failures while we were fixing the old ones. In the worst case we would never reach orange.

Once the colleagues give a green light on all tests, the merge goes in and we can sit and wait for bugs for our new feature to roll in.

Conclusion

I hope you found something interesting in this post. On a related note, SAP HANA is hiring right now in a few locations.

Discuss on Hacker News and r/programming.

C++ Quiz #1

2017-12-01T00:00:00-05:00

What is the output of this small snippet that is based on real C++ code?

#include <iostream>

struct Foo
{
    Foo() { std::cout << "standard" << std::endl; }
    Foo(const Foo&) { std::cout << "copy" << std::endl; }
    Foo(int) { std::cout << "int" << std::endl; }
    Foo(int, int) { std::cout << "int, int" << std::endl; }
    Foo(const Foo&, int) { std::cout << "Foo, int" << std::endl; }
    Foo(int, const Foo&) { std::cout << "int, Foo" << std::endl; }
};

void f(Foo) {}

struct Bar
{
    int m_i;
    int m_j;

    Bar()
    {
        f(Foo(m_i, m_j));
        f(Foo(m_i));
        Foo(m_i, m_j);
        Foo(m_i);
        Foo(m_i, m_j);
    }
};

int main()
{
    Bar();
}

The code compiles without warnings with clang++ 5.0.0 as well as g++ 7.2.0 using -Wall -Wextra.

Solution

The preliminaries are simple:

We have a struct Foo with a few different constructors, each of which prints a text that represents its parameter types.
We have a struct Bar with two members, m_i and m_j.
The main function calls the standard constructor of Bar.

The constructor of Bar is where the magic happens, so let’s go through it line by line:

f(Foo(m_i, m_j)) creates a temporary Foo object by calling the Foo constructor with m_i and m_j as parameters, so int, int is printed. The resulting Foo object is then passed to the function f.
f(Foo(m_i)) works analogously, it calls the Foo constructor with m_i as parameter, so int is printed. The resulting Foo object is then passed to the function f.
Foo(m_i, m_j) works the same as the first line, except it doesn’t pass the resulting temporary Foo object to a function, so it is destroyed again immediately, int, int is printed again.
So far so good, but now look closely. Foo(m_i) surprisingly behaves entirely differently from all the previous lines. It does not call the Foo constructor with m_i as the parameter. Instead it creates a Foo object of the name m_i, just like Foo m_i would. So standard is printed.
Now the last line looks just like the third line, but it still does something different. Why? Because the name m_i is no longer referring to the int m_i member of Bar. Instead m_i is now referring to a local variable of type Foo, so Foo(m_i, m_j) prints Foo, int.

Explanation

Thus spoke the C++ standard:

§8.3 Meaning of declarators

[…]

6 In a declaration T D where D has the form

( D1 )

the type of the contained declarator-id is the same as that of the contained declarator-id in the declaration

T D1

Parentheses do not alter the type of the embedded declarator-id, but they can alter the binding of complex declarators.

This kind of ambiguous parsing can lead to dangerous situations, as is documented in DCL53-CPP:

#include <mutex>

static std::mutex m;
static int shared_resource;

void increment_by_42() {
  std::unique_lock<std::mutex>(m);
  shared_resource += 42;
}

The code looks like it locks the mutex m while modifiying shared_resource, but instead a new mutex m is created, shadowing the global mutex m.

Future Directions

The upcoming Clang version will have a warning for this:

y.cpp:25:12: warning: parentheses were disambiguated as redundant parentheses around declaration of variable named
      'm_i' [-Wvexing-parse]
        Foo(m_i);
           ^~~~~
y.cpp:25:9: note: add enclosing parentheses to perform a function-style cast
        Foo(m_i);
        ^
        (       )
y.cpp:25:12: note: remove parentheses to silence this warning
        Foo(m_i);
           ^   ~