A Case for Generative AI?

I recently listened to “The Case Against Generative AI”, a four part podcast series by Ed Zitron. The Big Short remains one of my all time favorite movies, and I love a good bubble-story as much as anyone, but I’m also conflicted about rejection of the utility of Generative AI based on my own recent experience with it. So I wanted to gather some thoughts here, and try to be very specific. This text was written in one sitting, without using AI.

First of all: The economy of Generative AI seems to make no sense. It seems to try to defy gravity somehow, and a natural conclusion is to assume it’s immediate collapse. But many things appear to make no sense. Look at the stablecoin Tether — they have seemingly defied gravity for much longer with it’s 176 billion dollar market cap, and is still (seemingly) flying. So I won’t pretend to know if or when the whole, or parts of, the Generative AI market will go through a bubble burst, but I can say something about what it has provided me.

Now some context, feel free to skip this part: I am the co-founder and CTO of a software startup that delivers a comprehensive solution for data integration, modeling, analysis and reporting. The product consists of two repositories: one for frontend and one for backend. The frontend is made with TypeScript, React, Redux, Tailwind and Vite, and the backend is vanilla Java. Combined they consist of around 150k lines of code (not that anyone’s really counting), 98% (as of this writing) written by hand over the span of almost four years. I have had other engineers working on the codebase, but I estimate that my fingerprints are on 99% of the lines of code. With me in the company, I have my co-founder and CEO, as well as a COO who manages customer deployments and helps me with quality assurance of new releases. We had from the start a pretty good idea of what we where building, but almost all components have naurally morphed over time. Many components have been deleted entirely, some have taken on more and more capabilities, and some have changed to do something else than they were intended for. This has created, I’d say, a natural amount of tech debt, with some parts of the code are convoluted and more challenging to work with than others. To support refactoring, we have quite (again I’d say) solid coverage with unit tests and frontend tests with playwright. We have enabled strict mode on the frontend and “-Werror” enabled on the backend. So I feel quite confident when refactoring code.

As everyone else on the internet, I have been searching high and low for ways to use generative AI, both in the product, and in the business. A while back (~12 months?) I tried using claude to refactor parts of the frontend codebase, but to me it kept acting like a junior with no sense of risk. It kept breaking/removing subtle parts, which forced me to go through line by line to try to fix, probably spending more time than I would have doing it myself in the first place.

A few months ago, I had breakfast with Andreas Hennie. Andreas is a close friend of mine who also happens to be probably the most knowledge AI-person I know. I told the usual story about how AI has no idea of what it’s building. My go-to example is the 1201 and 1202 errors codes on Apollo 11. I can only imagine what I would have felt if I wrote the code that produced those errors. And what would the AI feel? It’s hard for me to trust code produced by someone or something that doesn’t feel the risk of getting it wrong.

Aaanyway, Andreas convinced me to make another attempt, this time thinking about it as me feeding the AI “the entire crime novel” as context, and then asking who the murderer is. “The AI will get it right every time”

Other things that had also changed since my last attempt, was the introduction of strict mode, having added “ — noEmit” to the tsc command, and much more comprehensive frontend testing with playwright. I’m sure the models also have gotten better, but that’s harder for me to measure.

Jumping a few months forward, there are two areas where I have seen real utility of AI in my work:

• Refactoring code
• Filling knowledge-gaps (I guess this one is pretty obvious, but I have a point about it later)

I started tagging PR’s produced by claude, and by now I have 9 merged PRs. The inserts in these account for roughly 4% of the frontend codebase. I want to explain one of them to explain what they actually changed:

Refactoring code

Data in our product is most commonly presented to the user as “objects”. Things, like car, customer, account, etc., that can be described with “properties”. In the beginning this meant that the user could look at data, but not really do anything with them. At one point a request came in from a customer to be able to generate a text from data that had been gathered from various sources. Without thinking too much about it, we added a simple textarea with some configurable text templates. A few months later, we realized that the little textarea probably was the most important feature in the product, so we decided to expand it to support charts, text formatting, dynamic filtering, and so on. But the report component still lived inside of the object component, and this became harder and harder to consolidate, as the two started having fundamentally different and often conflicting needs. I knew I should refactor the code to split the component trees and state, but it felt like too much effort for the little thing I wanted to change — and hey, look at that! more tech debt!

This challenge was the first thing I asked claude to fix for me this time around— to get the reporting component out of the object view, and separate it’s state and data loading logic. After some iterations, requiring very little mental effort from me, the build completed without errors, and playwright tests (which covers the object and report components extensively) passed. After some additional manual testing, and looking through the code for obvious smell, I merged the PR without modifications. Unconstrained by the object component, reporting is now much easier to develop further. The refactor has been running in production for about two months now. It’s hard to quantify how much value this produced, but it is some for sure.

The other PR’s have similar traits. Restructuring code to make it more readable, making components smaller, and centralizing computations like components heights. Some of them also introduced concepts I hadn’t yet gotten properly acquainted with, like memoization and custom hooks in React. This brings me to the second point, which is about knowledge gaps.

Filling knowledge gaps

I consider myself a capable programmer and IT person in general, but I know less about more things than I probably should. It has been clear to me for quite some time that Generative AI models are better TypeScript/React “code producers” than I’ll ever be. To make an attempt at an analogy, they are better than me at hammering nails into a wall, but they don’t seem to know much about how to build a type of house that has never been built before. Some specific “nails” that AI has helped me hammer includes the introduction of memoization and custom hooks, as mentioned earlier, suggestions for how to harden my OIDC implementation (“your honor, I would like to point out that it was already good, but it got better”), and integration with specific cloud APIs like Azure Log Service and Storage Accounts.

Say that I really had to fill those three knowledge gaps (though I didn’t), and I don’t have the time or energy to upskill myself (somewhat true), I would without AI need to hire people to fill those gaps. That would have required time and money to find, hire and manage. It’s also quite unlikely that I would be able to find an engineer that knew all the things I needed, so I would probably need to hire more than one. Given the nature of a startup, making such hires can be quite risky.

This leads me to think that generative AI produces outsized value for small teams like mine, but I wonder if the value becomes less significant the more people you have to fill knowledge gaps. Coincidentally, having more people use Generative AI would also drive up the cost, so wether this is an argument for or against the economic viability of Generative AI may be a little unclear.