Microblog

Yesterday I watched the Game Engineering Podcast with Animal Well‘s Billy Basso as the guest and it really gave me some new direction and got me a bit unstuck.

He said that that he is frustrated with modern games, because they start really slowly with lots of tutorials and exposition and they hold you hand so much that you never really get a chance to have a genuine discovery and insight, because they give you a hint before you even had a chance to look at the puzzle. He mentiones God of War, which is a sadly a very good example of this. The games don’t even give you a chance to get bored. Animal Well is really not like that at all, which is probably a huge reason why I enjoyed it so much, when I played it recently.

Of course he also talks about the technology and he mentions a bunch of cool things, most of which I could have expected from a handmade hero descendant. He says he just uses PODs without destructors or constructors, he mostly just zero-initializes everything with some inline initializers. He has a couple large blocks of memory that he preallocates at the start and wipes completely at certain points. There is a large block for fonts and vertex buffers and such, which have to live for the whole time the application is running, another block for the game itself, which lives for as long as you play and another block that lives as long as the current screen you are on. He said he never had a memory leak and doesn’t use pointers or anything, just fixed size arrays. Enterprise software has brainwashed me into disliking fixed size arrays and I think one of my biggest weaknesses is having disliked them for such a long time. Whenever he added a new type of object, he just added a new fixed size array for that object. First he just made up a number of the required size and later he did one pass through the screen to figure out how many of each object there are and then sliced up a big chunk of memory to fit all of the required objects exactly.

One really cool thing he mentioned is that he considers hot-reloading (both code and assets) one of the most important features of his engine and that this feature alone has saved him lots of editors and configuration files. This makes a lot of sense, because if you can just tweak the position of an object in the code itself, hit save and it immediately shows up in the game, you don’t really need an editor this much anymore. Having a data structure for some thing in the code is the bare minimum, but putting reflection on top, UI to edit that whole structure and stuff like that is usually a ton of extra work. You can just avoid it by having code reloading. I have been working on a Tomorrow Corporation engine thing for a few weeks now (including a giant blog post) and this really affirms some of my thoughts.

The way he does it though is simply having the game code in a separate shared library that is recompiled on demand and automatically reloaded. This really changed my direction regards to the Tomorrow Corporation prototype (more on that in the upcoming blog post).

A few other things he said that I liked are that he tries to keep the engine as thin and small as possible.

He also mentioned he wanted to double-buffer game state for a while so that the new game state is write-only and the old frame’s state is read-only (so your game update logic becomes a pure function), because this eliminates problems with order of execution, which are so, so common in video games. I did something like that for multiplayer games in the past, but I think with the new engine prototype I am working on I might put that in as a first-class feature from the start.

He also said that he does and uses the bad version of everything first. According to him most of the game engine advice will make you create a bloated engine with tons of stuff you don’t need. Most of the things that are considered essential parts of an engine are usually just required for AAA games with tons of people. You already don’t have many problems, simply because you work solo. And as a fun aside he mentions that he even put all the code into a single file until much later.

One thing that I will try to remember (continuously and have been trying to remember for years now): Don’t solve any problem until it actually confronts you as a real issue. Only make something more complicated when it hurts not to.

I am obsessed with Tomorrow Corporation Tech Demo for a while now and this weekend I had two cool insights that made me think it was possible to prototype this. The brief overview of features is this I would say:

• hot loading assets
• hot loading code on keystroke
• breakpoint, step forward AND backwards (aka time travel debugging)
• profiling with fast instrumentation
• find slow frames and functions and jump there (back in time)
• save timestamps into objects and jump to those stored timestamps
• seek backwards to any point during the demo
• portable sessions, saved every time

Some of the features are not that fancy or are simply a result of having a very fast custom compiler (hot-loading, compile in instrumentation for profiling). The real central piece of this engine is, I think a very efficient replay system. If you have a replay system that has little overhead and stores efficient sessions, you can record every time by default. Jumping to slow frames, saving timestamps and seeking backwards is essentially just a little feature if you have the replay system. This weekend I realized two important things. First, you don’t need to save every state of the program after every line of script code to be able to jump everywhere. Because the game is likely running at interactive framerates, you really just have to snapshot the state of the game every frame and in (probably) less than 16ms you can jump to every line of code, because it takes less than 16ms to calculate the whole frame. Second, stepping backwards is then also just a consequence of that, because you can simply restore the start of the frame and then simulate again, but stop one line earlier than last time. Usually when implementing time travel debugging it’s difficult to know when to do full snapshots, but in a game you have frames. So this is essentially easy mode! So what you really need is an amazing replay system.

You also need a scripting language that lets you debug it and control until where you want to execute. I thought I should try tinycc for this! And it was a massive fucking pain without any results. I time-boxed this experiment to this weekend and 80% of the time I was messing with tinycc. The most recent release on the web page, does not build on modern GCCs/libc anymore (no __malloc_hook). The current version on the dev branch has a different API and somehow I struggled a lot with making it find the runtime library, which imho should just be statically linked into libtcc. Then I wasted a solid 5 hours trying to get debug symbols out of it, which would be emitted if you built an ELF file, but not if you JIT compile to memory. I do not like the code. Anyway, I gave up. I looked into a ton of other things as well. QBE (can’t embed), cproc (need QBE), chibicc, lacc, fcc and pcc. I am flabberghasted that there is no little language you can embed from C(++) that JIT compiles and provides debug info. At least I can’t find it. And therefore this experiment is indefinitely postponed. MIR might be something worth trying once I actually bite the bullet and build my own scripting language for this, but that is neither today nor soon, because that would not be a weekend project.

Yesterday I attended a gamedev meetup and it got me really excited about making games again - not just engines and libraries. I decided I should take a break from the engine dev and make a game instead.. but with what engine? I hate them all. The solution I came up with was to make the game first and then go from the MIDDLE OUT (not the best fit, but I want to call it that now).

I’ll start with an executable that runs a Lua script, hardcode the window title, resolution - everything I can. Just add womf.newFrame() that returns false as soon as the window gets closed and clears the screen. Load a single shader from s hard-coded path and add womf.draw("modelfile.obj", {uniforms, ...}). The goal will be to build as little as possible of what could be called an engine and later on replace parts with nicer engine components.

I have been messing around with eBPF at work and I have become a huge fan of it. When I talked about this with my colleague at work, he mentioned a talk he gave at MariaDB ServerFest Berlin about eBPF very recently. I have seen a few resources on eBPF before but never quite grasped what kind of superpower it can be, before I used it myself. If I had seen Max’s real-life case study I would have understood much earlier how useful eBPF can be. So if you are developing or debugging on Linux and haven’t hopped on the hype train, this might sell you on it.

As a serial “new engine” developer I also have very strong opinions about what kind of game engine I would like to use or, more accurately, can tolerate using. Racenis has some ideas that I very much agree with. Especially the first three sections. I have a couple design docs scattered about that outline what I actually want from a game engine and I think I’ll turn it into a living blog post soon.

I think building things is bridging the gap between grand ideas of things you wish existed and reality. So a builder is constantly thinking of cool stuff they would love to see become real. Or rather someone that thinks like that tends to become a builder of some sort. And even though you spend so much time thinking about a cooler future, sometimes you get blown away by something someone else has done because you didn’t even consider that someone could think this big. The thing itself might be difficult, but not much more than a ton of other difficult things people do all the time and nonetheless people need to see the thing before they realize that it’s possible to do something like that. This video really blew the door wide open in terms of what I think engines and game tooling might be able to do. If I ever have a couple of years of free time and enough money I’d build exactly what Tomorrow Corporation showed in this video myself.

This is quite long, but very good. I think it’s part my “N+1 arc” mentioned in the post before. It introduces a really cool concept of tying the abstract concept of a lifetime to something concrete, which is allocation arenas.

Though he is very smart, I find Casey has a narrow view of software, because he seems unaware that many (most) software is not compute-bound, like games and adjacent software often are and he also shows a lack of understanding of the realities of larger development teams. He then tends to attribute apparent flaws in software today to people simply being bad or dumb, when in fact their practices are a result of a development culture that has good reason to exist, but not necessarily in (small) games. I agree it’s usually a mistake to misuse those common practices in compute-driven applications like games (e.g. TDD, heavy OOP, etc.), but people have good reason to do it this way, he just doesn’t really understand those reasons. Anyways my point in adding this disclaimer is that I understand perfectly if you do not want to watch this video or have to try not rolling your eyes as has as you can, if you do watch it, but it is worth seeing. I think this video and some other videos of his sort of started a new era for me. When I started programming I did it the “non-OOP” way, but in the most horrible spaghetti chaos kind of way. Later on I did learn about encapsulation, layers of abstraction and the value in making software robust, but I carried this into my personal, small (one-man) projects and it kind of ruined them. Everything was full of abstractions and allocations and crazy ownership graphs and just a lot of code that doesn’t actually do anything in the sense of modifying memory in a useful way. In large teams or old code bases the extra abstractions keep different programmers aligned, newbies in check and avoid subtle regressions, but a lot of this is likely unnecessary or counterproductive in some kinds of projects (compute-bound, small teams, etc.) or even certain pieces of code. Just when I started watching some videos of Casey I have been reading a lot of code by people that do have the “N+1 thinking” and I knew it was much better than mine, but I didn’t really understand why. The “N+1 thinking” is about structuring your code never around single objects, but always around collections of objects. Smart-Pointers, RAII, ZII? Becoming an N+2 programmer describes it in more detail, but the video is relatively long and somewhat noisy. Overall Casey opened my eyes on what exactly I should change and what I need to learn. I think I am still a little too OOP-brained, but I am currently on my “N+1 arc” because of him.

I have had “Website: Link Page!” on my todo for possibly years and this blog post inspired me to build this site. I post very sparingly on this website, because I don’t like thinking I have anything to say that anyone else has to hear, though I do feel like that sometimes. This also kept me from creating such a link page. Simon Willison’s post changed my mind about it - especially this paragraph:

The point of that article was to emphasize that blogging doesn’t have to be about unique insights. The value is in writing frequently and having something to show for it over time—worthwhile even if you don’t attract much of an audience (or any audience at all).

I really don’t really need a reason for someone to listen, when I am just screaming into the void anyways. Right around this time I also read What is microblogging? and I thought I’d go for something more general like that instead.