.png)
2 hours ago
2
I’ve wanted to write about this since the studio closed a year ago. Now that Contraband is also canceled I think it’s time, especially since Contraband was one of the big reasons why I left the company. The blog post turned out much bigger than expected though. There was a lot to get off my chest…
I worked at Avalanche Studios NYC from July 2012 to December 31st 2019, seven and a half years. I wasn’t there for most of Contraband’s development but it was obvious early on that it was going to be a very difficult project. If anything I’m surprised it lasted that long before being canceled.
The studio was born out of ambition. It failed because it could not deliver on that ambition. So this will necessarily be negative. But we had a good run and made two good games. I have so many memories and thoughts that I need to get written down somewhere, so lets celebrate the good and talk about the troubles.
Founding a Studio to Make Better Games
I wasn’t there for the founding. The studio opened at the end of 2011. It was an expansion for Avalanche Studios in Stockholm. (not to be confused with Avalanche Software in Utah) That company was doing well: Both Just Cause 2 and Renegade Ops were widely considered to be surprisingly good games. theHunter was a profitable franchise, and Mad Max and Iron Man (later canceled) were under development. The story I heard was that Christofer Sundberg (co-founder of Avalanche Studios) loved New York and liked the idea of starting a studio there. Then the stars must have aligned where they got a contract from Square to develop Just Cause 3 and at the same time were able to hire a team from Kaos Studios who had just shut down. One motivation was that the NYC studio would hire talent that they didn’t have access to in Stockholm, because apparently a lot of talent isn’t interested in moving to Sweden.
Also the studio had grown like crazy after Just Cause 1 to do several different projects at once, only for all but JC2 to fail. So I think they were scared to have three projects in one studio, so why not try starting a second studio?
Joining as a Junior Among Seniors
I joined as a tools-programmer in 2012, straight out of college. I remember being shown around the office and introduced to people as a tools-programmer and designer Jesse Johansen’s reaction was “oh thank god”, in a voice that made it clear how frustrated he had been with the tools.
The team was still fairly new. We had inherited an engine that went from Just Cause 2 to Mad Max to us. So a lot of Just Cause 2 features had been ripped out (water, animals) or had broken. When I joined, a basic version of water was working again but there really wasn’t much in the game. The most exciting thing was an art test where they had set up a small town with high quality buildings. (no one knew what “next-gen” would look like and the town looks significantly better in the finished game)
There had been no tools-programmer before, so the level editor had been kept barely working by random devs helping out a little bit, so productivity for designers and artists wasn’t exactly great. My first commit on my first day was to disable a “create road” button because it would just crash whenever you clicked it.
The studio had made a point of only hiring “senior” people. I’d guess a third of the team had moved over from the Stockholm office, another third came from Kaos Studios, and another third were from other places. I was the first exception to the “only seniors” rule as someone straight out of college. I felt flattered until I learned in my second week that the level editor I was supposed to maintain was scheduled to be deprecated and deleted because the “engine team” in Stockholm was working on a new level editor together with a new terrain system that would get the engine ready for next-gen hardware. (this would all fail and we would stick with the old editor for a long time)
The Old Editor Was a Mess
Learning that I would maintain dead-end software was a bit demotivating but there was plenty of work to do. When I started, the editor would crash 30% of the time on startup. When I tracked this down to some kind of graphics bug, the graphics programmer told me that he thinks Mad Max had a fix for this already and he would try to bring it over. While we shared code with Mad Max in theory, in practice the codebases were two separate copies that were constantly diverging. After Iron Man was canceled, Mad Max had two tools-programmers, so for a while I tried to bring over improvements that they had made, but this got more and more tedious over time as the code was becoming more different, so eventually it stopped.
The editor was also just in a terrible state. The 30% crash was the least of my problems. It was very difficult to make any changes to the editor because you’d constantly break some state that wasn’t kept in sync. When opening a “location” (the term used for anything that streams in together, like a town or a military base) all the objects in the location would be created, then deleted, then created, then deleted, then created and finally kept. This happened because different pieces of code in the editor didn’t trust each other. There were observers that watched for state changes and recreated objects, but these observers could be turned off sometimes for performance reasons, and then there was a backup call that would just recreate everything. And different pieces of code would trigger this backup, just in case, so it happened twice before you even made any changes to the location. And it didn’t get better when you actually started interacting with it. This took months of incremental improvements to untangle. It was obvious what the correct thing was (just create the objects once on startup and recreate them once for changes) but if you just stopped turning off observers, you’d get infinite loops or terrible performance. Whoever wrote the editor before had simply given up and piled on more bandaids, which just made the problem worse. (they were following the “make it work, make it fast, make it good” rule by undoing step 1 in step 2 and never getting to step 3)
I’m making it sound simpler than it really was. To illustrate the mess, know that there is a thing called “object-references” where things can refer to each other. (like “when the player enters this trigger, start that cutscene”) When the editor needed to know whether an object-reference was actually pointing at an object, it would walk the Windows tree view to see if an item existed at that position in the tree. Why did it ask the view instead of the data structure that created that view? Because the data structure (if you can call it that) was such a mess that you could not look things up in it. You needed to combine several different methods just to iterate it, and this iteration code was copy+pasted many times and was doing things incorrectly as often as not. And you certainly could not just ask if an object identified by a path existed.
Previous maintainers were also terrified of deleting code. The “create road” button that I disabled on my first day belonged to a road-system that had last been used in Just Cause 1. There was a new system where you create roads in “locations” through a different interface. The old system was still needed though because it knew how to flatten the terrain wherever you place a road, so there was a brittle set of steps you had to follow to convert roads from the new system to the old system. I made those steps not crash, then automated them, then got rid of all of the old road system except for the part that did the terrain flattening. I also ripped out several other big systems where I was certain that they had not been used in Mad Max or Just Cause 2. My lines of code for the first year were almost negative, which I was quite proud of.
All this is to say that I could see a microcosm of how Just Cause 1 and 2 ended up as buggy games.
The Game Was Almost Canceled
The game was coming along well actually. We were prototyping lots of things and the big important things (like the wingsuit) were working out. I can’t really comment on how decisions were made because I was too low-level. I think mostly this game went well because we had pretty clear direction (“next gen Just Cause 2”) with the occasional ambition that fit well. (“make crazy vertical military bases”) Sure, plenty of things were cut (underwater gameplay, multiplayer) but we kept the important things.
Also the right amount of restraint. Like the wingsuit was definitely overpowered at points during development. It still is a little overpowered in the shipped version, but you will smash your face into the ground several times before you get it right. Also we gave the player unlimited explosives but kept reloading of weapons. Need to draw the line somewhere.
The first big problem was when Square had financial troubles and almost canceled us. Somehow we avoided that fate by switching to being “free to play” (F2P) because that was a buzzword at the time. The idea was that the first island would be free and we would make people pay for extra islands and live service stuff, somehow. I have no idea how Roland and David got this through, but it kept the studio alive. Luckily Square’s finances recovered and that whole plan was scrapped. You can still see leftovers of it in the final game in that the first island is just the best island. The large island in particular was a victim of having not enough design due to being essentially canceled during most of the development.
I actually don’t have too much insight about the game itself. I was not involved in important decisions. When I boot it up now I just see all the bugs I caused. If you ever drive a car and get random unexplained bumps in the roads at intersections, that’s because road intersections required special handling (not my fault) and the tool for that was confusing (my fault). I could write a lot about why you don’t want to use Catmull-Rom splines for roads.
The New Editor and Terrain Were a Bigger Mess
Talking about my bugs means I have to talk about the new editor and new terrain. This was a giant mess. To the point where, when I left Avalanche, the new editor was going to be deleted again even though it had years of work by some of the most experienced developers in the studio. The new terrain was also a mess at first, but it could be saved with enough work.
I think the goal was that Avalanche Studios was going to license out its engine, like Unreal engine. But they recognized that the old tools were a mess and invested a lot of work into making new tools that were designed properly.
The new editor was written in Python. This was a huge mistake. The history is that, before I joined, the “location” compiler was rewritten in Python and the code was cleaner and actually ran faster than the old C++ code. So people thought Python was great and decided to use it for the new editor. I can see a case for wanting to use a memory-safe language for the editor, and I can see that Python is fun to write, but dammit Python is hard to maintain. I wasn’t there, but my theory was that the old location compiler was written by the same people who wrote the old editor. And obviously if you compare their code to a new implementation written in anything whatsoever, the new implementation will be better.
The code for the new editor was very designed. It followed lots of OOP best design practices. Everything was supposed to be a plugin. If you’re an experienced programmer you know how this went.
We hired a second tools-programmer, also a junior, whose main job was to work on the new editor once we started using it. He was constantly fighting fires. He was doing this so much that he was actually quite good at hiding how much of a problem we had on our hands. I knew, and I tried to keep my distance to the new editor. (this was easy to justify because there was enough work to do that for every month of work I finished, I would fall two months behind on my tasks) But I remember Andrew Yount (Tech Director) getting angry when he realized how this guy was spending his time. It’s weird to see Andrew get angry, that doesn’t happen.
People were constantly losing work. The new editor would crash and corrupt files, even late in development. Occasionally I would try to help out and see where the issues were and would immediately get completely lost in the layers. The way this thing wrote files to disk was so complicated that the devs thought it was a good idea if they took over version control (not a good idea if you’re also corrupting files). The team wasn’t that big, so in the old terrain you’d just make sure not to work on the same map tiles as other devs. The new editor tried to magic this away and failed miserably.
So our second tools-programmer spent a good amount of time recovering lost or corrupted files. Where in any sane environment you’d say “let me help you recover this file and then make sure this never happens again” half of his life was just “let me help you recover this file” because nobody could fix this editor. Unfortunately the fact that he was recovering the files also took the pressure off the engine team.
We had help from the “engine team” in Stockholm (who had written this editor) so they would actually ship features in the editor that we needed for the game. E.g. the new terrain had to be flattened for roads in a different way. This took much longer than in the old terrain because volumetric terrain is slower than height maps. But still this was interactive and nice at first. Then later in development it became slow and annoying to do because when you have a bad project it’s hard to keep nice things working.
Overall this whole effort was net-negative for development tools, and maybe even net-negative for JC3.
The terrain barely became shippable just before release. It almost caused us to run much too slow. I just booted up the game again on PS4 and we are nowhere near a stable 30fps. Is that entirely the terrain’s fault? No, there are other slow things, but the terrain alone was almost slow enough that we couldn’t ship, and required lots of effort that would have otherwise gone into optimizing other places.
The lesson there is to do one big rewrite at a time. Write a true 3D terrain or write a new editor. Don’t do both at the same time. Plus all the other lessons that ultimately caused the studio to fail, which I’ll get into later.
The Culture Was Actually Great
I remember the culture being pretty great. Great work/life balance and lots of young people together in NYC. The Swedes that had come over from Stockholm were great at having fun in the city. Also our address was on the corner of Broadway and Spring, which is a fantastic neighborhood.
I lived further out, in Flatbush. There were kids dealing drugs out of the front door of my building. Someone got shot outside my door in the first three months of moving to the city. I lived with a girl who was hyper-active and was constantly smoking weed to make up for that. I thought she was OCD and a little intense, but the girl I was dating explained to me that no, my roommate was actually crazy. Later I moved to Chinatown to a seventh floor walkup with five roommates who were constantly rotating. I lived there for three years and I had more than thirty different roommates in that time. It was actually great.
The development culture also felt vibrant. Cool things were constantly being added to the game. At the same time there was no pressure to stay late. When we had to do a bit of overtime at the end of development, the leadership felt so conscious about making sure that we didn’t work crazy hours, that the whole effort actually felt kinda pointless (I never worked more than 50 hours in a week).
The biggest issue is that things were also always breaking. Nobody but me wrote automated tests. (I had added the googletest library to our engine, which is one of the reasons why my “lines of code” was not negative in my first year) It was normal for the build to be broken for half a day. We also didn’t have any infrastructure around running older versions. If you synced latest and it was broken, you got to spend an hour however you like (clear out your email backlog?) until someone fixed the build. (this obviously didn’t apply to programmers who were expected to fix the build when they sync to a bad version)
At some point the company started giving out awards for doing well on certain criteria and I’m still mad that the first “craftsmanship” award went to a guy who was constantly putting out fires that he had caused. It’s not good to have a culture where you reward the guy who is causing the fires when he then puts out those fires. (this wasn’t the other tools programmer. There were multiple people who seemed to always be busy putting out fires)
But people saw how hard he was working on difficult bugs, and that was rewarded. Even as a junior I could tell that this was silly because there is clearly a better way of working where you follow better coding practices and don’t constantly write difficult bugs, and then have more time to do more productive things. Like Jacques Kerner (physics and vehicle programmer) steadily shipped one feature after another throughout all of JC3 and JC4 development. But that kind of consistency is almost boring, so it doesn’t get rewarded.
With that I want to call out some other contributors. I’m afraid I’ll forget to mention most. I remember Michael Knowland stood out as a fantastic character artist. Zach Schläppi was an art director with great taste but for some reason the team didn’t like him. (I never worked directly with him so I don’t know why) But in my mind he is the main reason why JC3 looked better than JC4. On design I remember liking Hamish Young because he always wanted things to be done better and had lots of ideas. (he was a bit of an “ideas guy” but his ideas were actually good) On programming Per Hugoson (my first boss) and Dave Barrett stood out as being able to write really difficult code that somehow never caused issues. Roland Lesterlin was probably a great game director in hindsight, but I didn’t notice it at the time. I’m still friends with Rob Meyer (though more because of JC4) and Clint Levijoki because they were great at what they do and they care about the whole project.
We Released to Good Reviews
JC3 was delayed a bit and released in 2015. Mad Max was delayed a lot and also released in 2015. This was a bit awkward because it created a bit of competition between the Stockholm office and the NYC office. Everyone in the company was pretty good about trying to not make it a competition, but it was unavoidable to notice that JC3 got better reviews. We felt pretty good in the New York office. I liked that John Walker (from Rock, Paper Shotgun) gave us their “RPS Recommended” award because I’ve been reading that site almost since it started.
I don’t know exactly what happened next but for some reason the leadership of the NYC office got fired because of some conflict with the Stockholm management. Roland Lesterlin and David Grijns had basically been the founders of the NYC office and their departure left a bit of a vacuum. They also took some devs with then but most people stayed. At this point I had been at the company long enough where I probably could have been in the know about what happened, but having started as the only junior on the team, this still felt like it was outside of my pay-grade so I only remember that there was “some conflict with Stockholm.” I heard that Stockholm didn’t appreciate the hustling where JC3 almost went free to play, and the NYC leadership didn’t like that they always had to save money. (e.g. free snacks were replaced by a vending machine)
I think both Stockholm and NYC could be a bit unyielding. Iron Man had been canceled because Stockholm refused to follow a too-tight schedule, and Mad Max had been massively delayed because there were disagreements about direction with Cory Barlog, which led to his departure and a big redesign of the whole game.
When you have such evidence of wanting things to be done a specific way, that’s going to clash with the NYC leadership who called their new studio “Defiant”.
Switching Things Up for the Next Game
I remember that Stockholm wanted to develop JC4 because they wanted their baby back but surprisingly we ended up with it again. Maybe Square really wanted to work with us again? And it superficially made sense that Stockholm would do Rage 2 after Mad Max, them both being desert games.
I was very optimistic for this game. The biggest issues in JC3 felt like they were externally imposed (going F2P for a significant amount of time, starting from Mad Max instead of JC2, the new terrain that caused so much work for so many people) and we were finally in a state where things weren’t constantly breaking. Just before shipping we had an engine where people could be really productive, and they were. We filled out that giant northern island in no time at all.
For me though I knew that I couldn’t do another round as a tools-programmer. This just wasn’t a studio that valued this work. For the entire development of JC3 we had a role unfilled for another tools-programmer. I thought the role desperately needed filling, but it wasn’t a priority. The “old editor” was still the tool used for almost everything and maintaining it really was way too much work for just me. Our second tools-programmer (who worked on the editor that could only do terrain) was going back to Stockholm, partially for health reasons but also probably because you can’t work on a terrible project like that for too long. If I stayed in my role I just knew that I would once again be worked very hard. It doesn’t feel good to constantly be falling hopelessly behind on all your tasks. Yes I shipped some good things, but I also knew that people were mostly unhappy with our tools.
As an example: I had written the “time of day editor” which was a tool for editing settings that change throughout the day. Like “color of the sky” or “fog thickness” or “wave height.” From things I had learned in college I knew that the best approach for something like this is to get a simple version working early, get it in the hand of users, and then iterate on the feedback. So I got my very first version out quickly. It was a bit buggy and had clear flaws but it got the basic job done, and it had one nice feature: hot reloading. Then I had to work on something else for a while but I was scheduled to get back to this tool in a few weeks. Then another thing came up and the time slot got moved back on the schedule a bit more. And then it got moved again. And again. And again. This tool stayed at version 0.1 throughout all of development. I got to squeeze in some small bugfixes by stealing time from other projects, and at some point a frustrated technical artist was able to help out a little bit to at least do the clear easy wins, but I never got to do the proper second pass that was needed on that tool. I was embarrassed when people used it. But when you’re sitting in a meeting to decide on the schedule and say “this tool really needs a second pass, it’s barely working” the answer is always something like “sure, but people can get work done in it and there is this other new feature that we really needed two months ago.” Which means you never get to do a really good job at anything.
Passing the Editor Torch
Between projects I had a bit of time to work on what I wanted, so I did two things:
- I shipped a testing framework that made it really easy to write tests that can run in the editor, so that things wouldn’t constantly break.
- I added a single Qt button. The old editor was written in MFC which was so old that Visual Studio wasn’t properly supporting it anymore. For the longest time I would repurpose existing menu items because nobody knew how to add new menu items or buttons. I figured that out eventually but the UI was clearly unmaintainable. There was an old project online that allowed interoperability between Qt and Windows MFC UIs. I got that working again and as a prototype shipped exactly one button. Just to prove that we could incrementally move to something more maintainable, like Qt. (it was also a great button: physics settling. A feature where you can select objects in the editor, hold the button and the physics simulation runs just for those objects. This allows the object to settle so that you can e.g. place a group of exploding barrels on uneven ground then hold the button for two seconds to let them settle. (It was a hotkey before it became a button))
This would in theory set up the editor for future success. We would have a more modern Qt UI and we would have tests!
I would move on to be an AI programmer. It had always been an interest of mine. And there was a slot freeing up because one programmer left to work at a different company.
I kept eyes on the tools work a little bit. A great tools-programmer (Viktor Arvidsson) had started in Stockholm and was driving the work to transition the editor to Qt. In fact the new terrain editor was going to be deprecated and the terrain-editing was going to be moved back into the old editor. The code had branched off from my state (as opposed to Mad Max) so I was happy that all my work of cleaning up was going to survive.
Sadly the “physics settling” feature broke pretty quickly and the editor tests were broken by a QA person who was trying to help out by doing some programming. I was aware of what his intentions were but I did not understand that he was submitting broken things. I only understood when the Stockholm programmer reached out to ask about the testing and I said “just run this thing” and then realized that it was completely broken and that I couldn’t easily get it working again.
Did I mention that Avalanche Studios didn’t really do code review? I’ll have to talk about that later. But it wasn’t that strange that somebody could submit something that was straight up breaking things that worked before. Since most people didn’t do testing, nobody cared that the editor tests didn’t work any more.
JC4 Tech Quickly Turned Bad
Anyway we were working on JC4. Starting off from a highly productive engine. It was going to be great. Sadly the engine became highly unproductive within months. People went back to their old habits right away. Turns out we were only stable near the end of JC3 because people mostly weren’t adding features. Most people were doing bugfixes or adding content or iterating on details. That makes it easy to keep things working. But for JC4 we were working on new functionality and immediately broke everything.
One disaster I remember was the gameplay scripting system. JC3 had shipped with two gameplay scripting systems. A “event” system which was obviously bad but was well-integrated with the engine. It led to constant bugs and because nobody could follow what even the simplest scripts were doing. And then we had another system that was inspired by Kismet from Unreal Engine. This system was written by someone on the Mad Max team over a holiday break. It was not well integrated with the engine so it could never possibly replace “events”. But at least you could see which objects were talking to which other objects and you could see in which order things were supposed to happen.
These led to so many problems and bugs in JC3 that it was an obvious point of attack for improving everyone’s work. As a result we wrote three and a half new scripting systems for JC4. If you thought that having two systems was a problem, wait until you ship with four and a half systems. Because “events” was the only system that was actually well integrated with the whole engine so we still used it.
How does this happen? Really bad planning and really bad design and not even properly thinking about the problem. Which can only happen because they didn’t have enough eyes on the proposals. (when I found out that we were doing the first two new scripting systems I immediately realized the problem and proposed a design for making “events” be graph-based to make it less error-prone, but my input came too late – the decisions had already been made)
Or we had a new “input system” to handle controls. It was outrageously bad. The person who wrote it left and nobody else could figure out how to implement remapping of controls. We almost shipped without being able to remap controls. I had continued to maintain my testing framework and was using it extensively to test AI behaviors and at some point I thought it would be a small additional lift to create tests for player behavior. But it was impossible, completely impossible, to send inputs from code. That might sound silly because clearly some code had to read the controller state and translate it into game inputs, but I could not figure it out at all. So I couldn’t write automated tests that control the player character for the silliest reason ever: because I couldn’t send inputs from code.
Other than the big screw-ups we actually improved lots of things. We were adding cool features to the grappling hook, we had more enemy variety, more vehicles, the game ran at a better framerate and had shorter loading times, it was easy to ship large content (the big city in JC3 was a major lift, in JC4 this was easy enough that we shipped several big cities) and ambitious features like the frontlines were done almost entirely by me, while they would have been very difficult in JC3.
JC4 Design Mysteriously Turned Bad
JC4 was weird because even though we improved on many things, the game came out worse. The design also had lots of improvements, going far beyond the “just blow everything up” gameplay of JC3.
When the game shipped I felt like something was off with it but I didn’t know what, so I thought we’d get better reviews than JC3 because we had improved so many things. The reviews disagreed, and reading the reviews clarified a lot for me.
The biggest problem was that where in JC3 the big bets worked out (wingsuit), in JC4 they didn’t. Like the “extreme weather” wasn’t terrible, but it also just didn’t carry a lot of weight. Also the frontlines never had any real design behind it beyond “these should exist.” I had implemented them but didn’t know what to do with them either so you can’t do much in them in the game.
The biggest bet that didn’t work out was the move away from “chaos objects”: The main criticism of JC3 was that it was repetitive. It was clear that we could never get great reviews as long as we didn’t address that. So we wanted to have more variety in the bases. And we prototyped so many different things for this. But it turns out to be impossible to design missions for Rico Rodriguez, the protagonist. Anything you put in his way can either be taken over or just walked around. How can you possibly design a mission if the protagonist can just fly over every obstacle you put in his way? Mostly we had two answers:
- Restrict the player to a small area. Either “hold this button without being interrupted” or “don’t leave this zone for 1 minute.” I don’t know how many of these missions we shipped with but I played way too many of these during development.
- Escort Quests. If you can’t restrict Rico, make him protect someone who is restricted. Now you have to clear all the obstacles for NPCs.
Players hate escort quests. The AI team (which I was now a member of, remember) hated escort quests. You can’t have escort quests in a Just Cause game because there is constantly debris everywhere. There is always some car wreck or a broken giant fuel tank blocking the way. Early in development the designers on our team made it clear to the rest of the design team that we would not give any support to escort quests. Our message was: don’t make these. And yet half the game is escort quests and we were very busy supporting escort quests. I think the other AI programmer on the team, Anders Südow, still has nightmares because one early mission involves escorting a car half-way across the map while it’s being chased by AI enemies and while there are roadblocks in the way. And it’s mostly systemic, only the road blocks are scripted. A crazy amount of work went into that mission because (surprise!) it’s not good to block the road that AI is supposed to drive on. And the mission is not even all that good.
How does this happen? A decision was made and we stuck with it. We weren’t going to do chaos objects any more. At first there was a notion that if things didn’t work out, we could always fall back on chaos objects. And then we never did that when things didn’t work out. Instead you get lots of missions of “go there then press the button.”
Maybe nobody realized the fundamental impossibility that you can’t design missions for Rico Rodriguez. I certainly didn’t put it together until after we shipped. (but it also wasn’t my job)
I personally liked our game director, Francesco, quite a lot. He was the lead designer on JC3 and he is clearly a good designer. But he is a bit too technical in that he likes to come up with systems and rules in the game. Somehow he missed that the basics were often not there. Like these missions aren’t fun. And a game director is also supposed to have his eyes on other things like graphics (characters look worse, water looks worse) or story. Watch a random cutscene from JC3 and it’s people having fun and is short and to the point. Watch a random cutscene from JC4 and it’s not people having fun and it’s not short and not to the point.
The main thing I remember him doing was managing Square, but I don’t actually know what that involves.
Nobody Ever Gets Credit for Fixing Problems that Never Happened
My main issue was that things were constantly broken during development. We’d ship some new feature to Square for a “milestone” and then a month later it was broken and nobody cared to get it working again because we were busy shipping the next “milestone.” Usually you couldn’t play missions anymore that were made two months ago. Only in the last months before release did everything actually work and stay working.
I was still the only person writing automated tests. There was good infrastructure for this in place now. I remember giving a talk internally where I tried to convince people to write more tests. I asked before the presentation “who already thinks we should write more tests?” and almost everyone’s hand went up. Which left me a bit surprised because what the heck are we doing then? In practice it was completely impossible to get these people to write tests even if they thought they should be doing it. One time there was an animation bug that was causing issues for me. So I wrote a little test to reproduce the issue and I showed it to the other programmer. “Look how easy this was to set up” and “look, you can restart the test from the menu or run it on repeat to make it super easy to debug the animation issue” I’d say. And he’d nod appreciatively, fix the issue (two months later) and then never write a test of his own.
I was reading a lot of project management literature at this time because I couldn’t figure out what was going wrong. The problems were so obvious. The solutions, too. Just take a moment and think about what you need for gameplay scripting before you write three and a half different systems that all don’t solve the issue. Just take like ten minutes to write an automated test so that a new feature doesn’t break in a month. Invest a little bit of time to save a lot of time.
I found the answer in the classic paper “Nobody Ever Gets Credit for Fixing Problems that Never Happened“. Man were we ever stuck in a capability trap. Or a Red Queen’s race: You have to run as fast as you can just to stay in place. Here is a quote from the paper that could have applied perfectly to Avalanche Studios:
“Supervisors never had time to make improvements or do preventative maintenance on their lines . . . they had to spend all their time just trying to keep the line going, but this meant it was always in a state of flux, which, in turn, caused them to want to hold lots of protective inventory, because everything was so unpredictable. A quality problem might not be discovered until we had produced a pile of defective parts. This of course meant we didn’t have time to figure out why the problem happened in the first place, since we were now really behind our production schedule. It was a kind of snowball effect that just kept getting worse.”
You see we never had time to keep things working because we were behind on shipping the next milestone because so many things were broken.
I remember, years later, talking to another programmer who had been on the same team. And when I talked about testing he said something like “Avalanche was always doing big games with a small team. We just didn’t have enough programmers to have time to write tests.” This blew my mind. It explained a lot actually. People thought that if you want to write tests, you have to hire more people so you have more time to write tests. For me it was always obvious that tests save you so much time because you’re not constantly breaking things. Instead of being the guy who is constantly putting out fires, you can be the guy who steadily improves things and actually ships features. Meaning a team who writes tests needs fewer programmers to achieve the same thing.
But it wasn’t just the programmers. Everyone was constantly behind on everything. There were many things in JC4 that were actively worse than in JC3. Like more things were destructible in JC3. Much of the art is worse, some environment art was ridiculously bad. And yeah, it’s kinda silly that people on reddit were making fun of us for having blurry low-res low-poly rocks in the game. Because who really cares about rocks? But it was just a symptom of this wider issue that can be felt all throughout the game. And when you have a clear example, it makes sense to highlight it.
When there are constant issues all throughout development, nobody gets to ask for really high quality things. At the time I read the literature and watched the talks and found all the quotes: John Romero talking about the early days of id software and how they would constantly fix bugs as they saw them, Rob Pardo talking about how Blizzard doesn’t just polish at the end, you polish all throughout development. Read that linked paper to see how process improvements can drastically improve output in other industries. Here is another quote that could have applied to Avalanche:
a BP team reduced butane flare-off to zero, saving $1.5 million/year and reducing pollution. The effort took two weeks and cost $5000, a return on investment of 30,000%/year. Members of the team had known about the problem and how to solve it for eight years. They already had all the engineering know-how they needed, and most of the equipment and materials were already on site. What had stopped them from solving the problem long ago? The only barrier was the mental model that there were no resources or time for improvement, that these problems were outside their control, and that they could never make a difference.
Avalanche had a couple of these lying around. E.g. I tried to convince people to implement the “not rocket science” rule of software engineering which is to “automatically maintain a repository of code that always passes all the tests”. We had almost no tests, so I would have been happy to automatically maintain a repository that builds. This is not difficult (it’s called “not rocket science” for a reason) it would have taken a few days of figuring things out in Perforce and Jenkins, and it would have paid for itself immediately, but we were too busy.
And in that culture you just cant ask for the really nice things that actually make a game good.
How Obvious Physics Bugs Ship
Here’s a fun one for learned helplessness: When John Walker posted his JC4 review it was clear that he picked up on all the issues. He pointed out so many things that were not only bad, but pointlessly bad. Like how the game always shows the controls for honking the horn when in a helicopter, and there is no way to turn that off. And at the same time it doesn’t tell you how to actually fly the damn thing. (how does that happen? It got submitted somehow, maybe as a first version, and it wasn’t bad enough that it required someone’s full attention, and it was related to the bad new input system which had a big “ugh field” around it, so it stayed)
But he also posted a video with a bunch of physics bugs. I looked at that and thought “that looks worse than it should.” Sure you could break our physics, but it shouldn’t be that broken. I thought the JC4 physics was generally better than in JC3. I had an immediate suspicion though. I went into work the next day and found the code for clamping physics impulses.
Any physics engine has problems when you present it with impossible situations. Like a car inside a rock. It really wants to push the car out, but since it’s an impossible situation, it is going to apply crazy forces. And then those crazy forces cause more impossible situations to happen so things can never calm down. And in JC4 it’s easy to cause impossible situations because we give you lots of tools to mess with the physics. To solve this we clamped all impulses that the physics engine applies to some reasonable intensity. Meaning it can’t push the car out in one step any more, but things are more stable overall. I looked at that code and found that it was not that complicated. It just had some if/else statements to check if an impulse was too big, and to log the crazy impulse during development so that we can identify what’s causing them. Except the if/else was wrong. It was supposed to disable that logging in release builds, but leave the clamping enabled. It disabled both the logging and the clamping. Meaning only in the released game could you get unclamped crazy physics impulses that we never saw during development.
I walked over to Jacques, our physics programmer (who was usually great, see above) and we had it fixed in less than five minutes. It made it into the game in the next patch, but too late for reviews.
How does something like this get through QA? How can they not catch that the physics can go completely crazy in the released game? My guess is that QA did catch this and there were probably several entries in the bug tracker for crazy broken physics. But remember how things were always kinda broken during development? We probably had hundreds of bugs for broken physics in the bug tracker. Most weren’t as bad as the things we saw in release builds, but nobody could quite tell what level of badness was acceptable and what level wasn’t. So we shipped a game where you regularly saw physics go completely crazy.
The DLC Was More Fun to Develop Than to Play
In JC3 I didn’t work on the DLCs, but in JC4 I did. Once again this was better because the engine was now in a good state, not constantly broken. When you work on the DLC you have to make absolutely sure that you don’t break the main game, so people are more careful and don’t constantly break things. So everyone is actually productive.
My favorite part of working on JC4 was the “Getting Over It” easter egg. Rob Meyer is friends with Bennett Foddy, who lives in NYC so we got him to come in. For the Racing DLC I made a second bigger version of it. It’s great, even though almost nobody has played it. While writing this I discovered that someone finally uploaded a video to Youtube a year ago.
The concept of the Demon DLC was “bring back chaos objects.” Which is a great idea. I did all the enemy AI. Sadly the DLC didn’t really work out. Since I worked on the enemies, and you could probably call me the main combat designer, I mostly blame this on the combat. Enemies keep on respawning, so many players adopt the strategy of “try to clear the base even though I’m still being attacked” which means you end up playing a game where you’re constantly being pestered by the enemies, which is just unpleasant. The best way to play the DLC is to keep on killing the enemies because they eventually switch to a mode where they respawn more slowly, at which point you can do the chaos objects, but players don’t realize that. (I had to hide this “slow respawn” mode a little to get it through) We saw the problems during internal playtests and I didn’t want to do the infinite respawning, but I was overruled because it felt weird for the towns to be empty. So once again a decision got made, we saw it was bad, and we stuck with it because we couldn’t come up with anything better.
The hoverboard DLC was great fun in that it solved the design problem we had in the main game: Turns out you can design varied content for Rico Rodriguez by reframing them as chaos objects. I didn’t work on it for long though, and my main contribution is that I added the really long hoverboard tracks. I also worked on the boss fight though, once again, I am apparently not a good combat designer because that boss fight is not that great… (I’m better as a programmer)
I tried to internally pitch that we should make a hoverboard racing game, but it was rejected because the NYC studio wasn’t supposed to work on small titles.
Lack of Faith Made me Leave
I never worked on Contraband since I was busy with JC4 DLC and a bit of Rage 2 DLC. I saw early prototypes of Contraband and heard all about the design, and it was clear to me that it was going to be trouble.
I didn’t want to work on another troubled multi-year project, so I left before joining the team. Another contributing factor was that a bunch of my code was lost, including all the work I had done on automated testing. This time the engine would continue from Call of the Wild, so if you wanted to keep something, you had to merge it into that code base. I had actually convinced people that automated testing was important, and apparently there was a meeting to decide what to do about it, but they decided to go with a different, less fleshed-out system that someone in Stockholm had recently written. It wasn’t a bad system, I just thought they made it unnecessarily hard to write and run tests, and it clearly was at best a first version of testing infrastructure, where my work had actually been used extensively and shipped in a game. But I didn’t get to make my case because I wasn’t in the meeting. Yes, they decided on the future of automated testing without involving the one person at the company (in both NYC and Stockholm) who actually had experience of doing extensive automated testing. I’m still bitter about that… I had briefly thought about adding all the nice features that my testing infrastructure had to the new system (like writing tests entirely in content using “events”, which I had actually convinced one designer to do; or being able to pause tests in the middle to debug something; or running a test on repeat), but the new system was so far from supporting nice features that my motivation quickly got deflated by how I would not only have to redo my work, but would have to put in extra work because the design of the new system wasn’t there yet.
If there’s one lesson in this, it’s that not sharing code and throwing away code at the end of a game can really hurt your engine, both in all the obvious ways and also in several non-obvious ways.
Contraband Was Going to be Very Very Difficult
I still saw lots of Contraband before leaving. The company had pitched a lot of different projects, but Contraband actually got signed. Contraband was a lot of things to a lot of people. It was going to be the justification for having a NYC studio, going back to the original motivation from 2011: A big impressive AAA game that justifies having an expensive studio. In fact the game was big enough that NYC and Stockholm would work together on it, somehow.
It was going to be a multiplayer co-op open-world heist game. The first three things made total sense: We could see how much money GTA IV was making and we had wanted to make a multiplayer game for ages (JC3 had some work done towards multiplayer in the F2P era). Co-op makes sense for that because it’s easier to make a co-op multiplayer game than a competitive one, especially in an open world. Open-world made sense because that was our strength as a studio. But what the heck is a heist game? I think the idea was to appeal to a similar audience as GTA so it had to be more grounded. The proposed gameplay sounded a bit like Hitman where you have to scout out a location and then steal something. Or maybe like Teardown, though that wouldn’t come out until years later. There were also talks of smuggling using truck convoys, both driving them in co-op and trying to steal things from AI-driven ones. There were neat ideas but I don’t think anyone actually understood what the core of the game was.
So I don’t know what the game was trying to be, but if it wanted to be a multiplayer open-world Hitman, that sounds good in theory but I don’t know a single studio in the world that could pull that off. It sounds really really difficult. Even a single-player Hitman is really really difficult and requires very strong designers, which we had just demonstrated with JC4 that we don’t have.
The game director on the project was going to be the former narrative director from JC4. This made no sense because the narrative was not good and in fact was a bit mismanaged (though we had problems with a vendor, and things could be blamed on that). The reason he ran the project was that somebody from the NYC studio had to lead this game because it would have sucked to just be a subsidiary helping out Stockholm, but Francesco had just been game director on JC4 which wasn’t good, and nobody else made sense either. The person who eventually was the lead only made sense because everyone liked him. And he was really good at selling things. So he pitched a great game concept to Microsoft and got the project signed. (and also no project pitch from the Stockholm office got signed, so there were no alternatives)
I remember talking to him explaining that the game he was trying to make sounded really difficult from a gameplay perspective. Especially how it wasn’t supposed to be a shooter. Like everyone knows how to make a racing game because racing is inherently fun. Or a platformer or a shooter because platforming and shooting are fun interactions on a computer. And while this game had bits of driving and bits of shooting and maybe even bits of platforming, it wasn’t going to be about any of those things and I couldn’t see which part would be inherently fun. The various ideas that I had heard all sounded really difficult, like they weren’t going to work out. And after they wouldn’t work out they would be replaced with shooting, which would have gone against the design. The response I got did not make me think that he appreciated just how difficult of a project he was taking on.
I only know how it went from second-hand reports of people who stayed longer. Apparently they kept on trying new things saying “now the game will be this” for six months, then when that didn’t work out they said “now the game will be like that instead” for six months. I don’t know if they ever got out of that mode.
Apparently there were negotiations about starting development on another project, but that fell through. So eventually the studio shut down.
It’s All About Culture, in a Wider Sense
In my mind this is all about culture. You couldn’t fulfill the ambition of the NYC office with the culture of the NYC office.
To explain what I mean by that, let’s talk about countries for a second. The main reason why China in the 21st century is doing much better than China in the middle of the 20th century is culture. I mean culture in the Joseph Heinrich sense. So not just “books” or “music” or “art” but also things like “how companies are run” and “what you learn in school” and “what politicians care about” and what I talk about in the “culture” chapter above, as “work/life balance” and other unmeasurable things. You can invoke lots of other reasons but the culture of China in the middle of the 20th century was very limiting. You just can’t become a top tier country with that culture.
What specifically is holding you back? Lots and lots of little things, and then a few big things like the Great Leap Forward. But even without that, the lots of little things will just constantly drag you down. They’ll act like a mix of gravity and friction, keeping you from the top tier countries.
This is also obvious if look at other examples. You might compare Denmark and Afghanistan and say that Denmark got lucky with Novo Nordisk and Afghanistan got all kinds of unlucky. But really the main difference is culture and all the millions of little things that Denmark does right and Afghanistan does wrong. Afghanistan is being dragged down by its culture and many other issues could be overcome with better culture.
When Avalanche Studios NYC was founded, Avalanche Studios as whole wanted to step up to the next tier of game developers. There were big investments to make the engine ready for next gen, maybe even license out the engine. Use that money to reinvest and then grow the studio. But we didn’t have the culture to level up. I don’t think Avalanche Studios Stockholm had it either.
There are also no secrets about how to do this. The paper to read here is “The Mundanity of Excellence.” It talks about excellence in the context of olympic swimmers. Having now worked at other places, I think its thesis for how excellence is achieved is spot on:
Excellence is mundane. Superlative performance is really a confluence of dozens of small skills or activities, each one learned or stumbled upon, which have been carefully drilled into habit and then are fitted together in a synthesized whole. There is nothing extraordinary or super-human in any one of those actions; only the fact that they are done consistently and correctly, and all together, produce excellence. When a swimmer learns a proper flip turn in the freestyle races, she will swim the race a bit faster; then a streamlined push off from the wall, with the arms squeezed together over the head, and a little faster; then how to place the hands in the water so no air is cupped in them, then how to lift them over the water; then how to lift weights to properly build strength, and how to eat the right foods, and to wear the best suits for racing, and on and on. Each of those tasks seems small in itself, but each allows the athlete to swim a bit faster. And having learned and consistently practiced all of them together, and many more besides, the swimmer may compete in the Olympic Games. The winning of a gold medal is nothing more than the synthesis of a countless number of such little things – even if some of them are done unwittingly or by others, and thus called “luck.”
I left video games because I was looking for a place that can ship high quality software and found it outside of video games. (I would have stayed in video games if there had been such options in NYC) Having now worked at a place that has a better culture, there really are no tricks. I think the new place writes just as many bugs as Avalanche Studios NYC did. But the rest of the culture means the bugs don’t cause as many problems. Most of them get caught before being submitted to main, and the ones that do make it through don’t cause as much damage because there are layers of defense. It just takes lots of small activities and “the fact that they are done consistently and correctly, and all together” as in the quote above. My advice for Avalanche would be to do small things, like write automated tests, do code review properly, actually share code between projects instead of copy-pasting the engine and diverging, run the autobuilder before a change gets submitted to make sure the build can’t break, do postmortems when there was a big enough issue that others noticed and actually do most of the follow-up actions that you agreed on, be curious about bugs when you run into them and try to fix them proactively. No magic, just small improvements done consistently for a better process.
I remember wasting days at Avalanche on trying to track down this weird crash. It was something about the GPU doing delayed work and crashing in a spot where you couldn’t see what originally caused the issue. Weeks later a technical artist was also trying to track this one down and together we made some good progress and finally found out “it must be something with X,” where X was written by one of our graphics programmers. We talk to him and he says right away “oh yeah X crashes sometimes.” He had known. And just left it in. And this guy wasn’t even the bad guy I had talked about before who was causing all the fires. Nothing happened as a result of this episode. The bug got fixed and that’s it. (this is obviously crazy, but at Avalanche Studios this was so normal that I have to say what to do instead: at least have a post-mortem about how this could have happened. Then decide on actions to prevent this from happening again and do those actions)
With the culture that Avalanche NYC had, you could either throw more money/people at the problem or work crazy overtime, or you could be stuck at a certain quality level. We didn’t have lots of people and we didn’t want to work crazy overtime, so we were stuck. And the companies that just throw more money at the problem are also a bit stuck. When things break at the same pace that you’re fixing them, you end up with some parts of the next game being better and some parts being worse, but ultimately you end up at the same quality level.
I think JC3 was our limit. It was a better version of JC2, and that’s all we could do. In JC4 we had ambition to do better than in JC3 and as a result we ended up in a Red Queen’s race where we just couldn’t make progress and actually ended up worse than where we started. Contraband wanted to be even better and as a result just couldn’t ship.
I actually think the company strategy was kinda smart. I think it made sense to expand to the US and to develop the new terrain at the time when they did. I think it made sense to try to redo the terrible editor code and to try to license out the engine. It made sense to go multiplayer for Contraband and to do it as a co-op game. But then there was lots of bad execution which led to us constantly being in trouble and constantly being behind, which led to us sticking to bad decisions.
I am also worried about the rest of Avalanche Studios. They were growing a lot when I left, and were working on several self-published titles like Second Extinction and Ravenbound and The Angler. Those games have shipped, to various levels of disappointment (I haven’t played any, but if I had to guess, The Angler might actually be good, the others not), and I haven’t heard about anything since. It’s not good that it’s been almost six years and they have only shipped some middling games that were already in development six years ago. (and I literally have no inside information about what they are working on) And also the track record wasn’t great before that. Rage 2 didn’t get great reviews either, and neither did theHunter: Primal or Generation Zero. They weren’t terrible games, just disappointing compared to what they could have been. And internally there wasn’t much reflection on that, the official messaging was always “they made their money back and were a success for us” which isn’t that satisfying.
Lessons Learned
Speaking of which, lets get to the lessons we learned here. One of them is that you can’t develop games any more like we did for old console generations. And one of them was that Avalanche was not good at learning lessons. It’s not like they didn’t react to mistakes, but they often drew the wrong conclusions. Lets get to these in order.
Short-Lived Code
Games used to have development cycles where it made sense to not write super high quality code. When going from SNES to Playstation to Playstation 2 to Playstation 3 you were probably going to rewrite much of your engine anyway. This stopped being true with the Playstation 4, just as I joined the company. Not just because you have to live with the same code for multiple projects, but also because each project takes longer and has more code, so the crappy practices bite you before you even ship the first game. Avalanche was stuck in the old mindset for way too long. Programmers as a craft know how to do this better, but game developers were very reluctant to adopt practices that slow down individual changes in order for the project as a whole to move faster. They like to point out how things also are done badly in other industries (Slow buggy web apps, OOP design-patterns, micro-services) and as a result they reject the successful practices, too.
It was also clear that others in the games industry had the same problem. Mass Effect: Andromeda came out the year prior and got panned for bad character art and bad animations compared to earlier Mass Effect games (from earlier console generations). Or look at that comparison video of Far Cry 2 and Far Cry 5, which came out the same year as JC4.
What can be done about this? The easiest thing is to stop having your own engine. That limits how much damage your programmers can do. The more difficult thing is figuring out how to learn the right lessons.
Postmortems
Going back to that “culture of countries” comparison, while we obviously didn’t have anything nearly as bad as the “Great Leap Forward”, it’s helpful to think about how such a colossal fuckup can happen. It’s almost calming to see that a country that can evidently be very competent can also have one of the biggest screw-ups in history. But thinking about it, I think one essential ingredient for messing up really badly is that you don’t learn lessons.
We actually did try a bunch of improvements. We tried code review for a while but we did it badly and eventually it fizzled out. We kept on trying to share code between games and we kept on doing it badly, causing us to do very little of it. I did convince some people to write automated tests, but they would write one or two bad tests and then give up. Remember that QA person who was ruining the editor tests? He also wrote some other tests that were worse than useless. They would break all the time on nonsense issues and wouldn’t actually tell you what the problem is. This is actually normal when you first start writing tests, because it’s a bit of a skill. So you need to learn your lessons, delete those tests and write better ones. This never happened.
And when we did learn lessons, it was non-public and it was not clear how we came to exactly those lessons. E.g. the “old editor” was clearly a problem when I joined and that was the reason why there was going to be a new editor. The lesson they learned was to not write it in C++ and to write it in Python instead. This certainly was going to decrease crashes due to memory-safety issues. But would it fix anything else? And wouldn’t you increase crashes due to lack of type-safety? In hindsight the correct thing was to instead do the tedious work of slowly fixing all the issues with the editor over a couple years.
Coming to that conclusion earlier is difficult, but I know how to not do it: If you don’t have a postmortem, you certainly won’t learn from last time. If there was a postmortem for the Python decision, it was never shared with me, and I was the only other tools-programmer on the team, so that would be very strange. I remember walking towards lunch with Linus Blomberg when he was visiting NYC and he was very reluctant to talk about the whole thing. He certainly wasn’t going to write a postmortem. And if you don’t write a postmortem about bad decisions, you make more bad decisions like writing multiple new scripting systems that all didn’t solve the issue. And there were no postmortems shared for any of those either.
Even if people “learned their lesson” from these mistakes, who’s to say that they were the right lessons? Part of the value of a postmortem that’s widely shared is that you get lots of comments on the document and people will tell you all the important things you forgot. (as I expect will happen with this blog post, because my view is necessarily one-sided. E.g. I remember designers being frustrated that even if we learned lessons from playtests, we didn’t act on them. But that has to be someone else’s perspective)
Reviews
Another aspect of this was the lack of serious review. Meaning code review, but I’m not just talking about programmers, but also review for designers and artists.
I now spend more than 10% of my time doing review. It easily pays for itself. When it gets to be more than 30% I try to reduce my review obligations, so the right number is somewhere between those two.
Code review done properly helps with so many issues that we had. Your code has to actually be up to a certain standard to be possible to be reviewed at all. I remember people submitting things that were just straightforwardly wrong. Like multi-threaded code where a background thread writes to a data structure and the main thread reads it, and there is no synchronization between the two, at all. When reviewing multi-threaded code, if the synchronization is not clear, I leave a blocking comment that forbids release. If it’s too confusing to review, it shouldn’t be submitted.
I remember talking to a designer who had left Avalanche to work somewhere else, and hearing his appreciation that the new place challenged him, really, to explain why an idea would be fun. I remember hearing the description for a mission in a weekly demo of new features, and all I could think was “but why is this interesting?” I don’t think the designers at Avalanche were really challenged to explain why things were good.
I don’t know the process for artists, though I’m pretty sure some of the bad assets that made it into the finished game were not reviewed.
Review as a practice just helps all around. Spreads knowledge and responsibilities, raises standards, prevents heroes or leads from acting on their own. Just make sure that every change is reviewed by at least one other person and that it doesn’t get released until the review is finished. Then iterate on the process until everyone’s productive within those rules. (e.g. if you put a change up for me to review, it should be easy for me to run your version of the build)
Thesis
My overall thesis here has been that we were hampered by bad development practices and couldn’t achieve our ambition. If you have followed Avalanche Studios and looked at all their games, this might sound obvious. The development on the inside was exactly like you expect it to be from what you see in the shipped games. But I don’t think this awareness existed on the inside, or if people were aware they saw it as an unchangeable fact of life. If I had to guess, internally the number one reason would probably have been small team sizes and budgets compared to the size of game we were making. Which is a dangerous justification because sure, Generation Zero (random game developed by a small team) would have been better with a team twice the size, or with twice the time, but you don’t get to have twice the team size if your previous game was no good. You can either be stuck in a never-ending cycle of mediocrity, or you can figure out how to level up.
Video games as an industry will eventually figure this out. Just by evolution. Studios with bad culture will die out and studios with better culture will survive a little longer. If I ever get back into gaming I would try to optimize for productivity. Games don’t have to be as robust as the software running on the Mars rovers, and aiming for that would be a drag on productivity. But if something easily pays for itself, like having a build that’s not broken, or having automated tests for the main features of your game, then you should prioritize that. Don’t even get close to a capability trap. It makes sense that the tools programmer thinks productivity is a top priority, but it made no sense that everyone else always had something more important to do.
