Pair Programming with ChatGPT: An Experience Report

For a very long time I’ve held off consulting Large Language Models besides the odd query out of curiosity that fizzled out rather quickly. But this week, I broke my own rules and pair programmed with ChatGPT twice. Here’s a quick summary of my findings so far.

Everyone else at work was on vacation and I had to re-implement a specific feature from Java framework 1 (Spring Boot) to Java framework 2 and 3 (Quarkus and Micronaut) with zero knowledge of the latter two. After combing through the obligatory guides and documentation, I quickly realized that the specific thing required more than a hello world Controller endpoint with a few @Injects thrown in for good measure. For instance, I wanted to mimic a certain aspect-oriented thing from Spring in Quarkus, and since the latter is bound at compile-time instead of run-time, you need to approach things differently.

Anyway, the implementation details are irrelevant and a bit boring. It surprised me how quickly ChatGPT is able to whip up some kind of example that initially seemed to satisfy most of my requirements. Just to be clear: I did not blatantly copy-paste the code and asked it to convert it for me. I asked it how specifics under the hood work so I can have a better understanding of how to approach the problem.

The code it provided did not compile.

ChatGPT insisted that @Interceptors that are able to catch method calls with @AroundInvoke do not need the annotation that denotes which methods you want to catch to be marked with a specific (Jakarta) annotation as well. As far as I understood things after reading the CDI reference, and according to the compilation error, it did. Long story short, ChatGPT was wrong, and I pointed it out.

It apologized, reframed the solution using a completely different approach that was unusable for me—which it knew based on our previous conversation. When my response then is something along the lines of “but this approach won’t work, I already told you that”, it apologizes again, corrects itself, and presents the wrong @Interceptor code—again.

It seems that we’ve entered a deadlock. I reframe my prompt, it proposes solution A, I tell it that is not valid, it proposes solution B, I tell it that won’t work because of our constraints, and then it goes back to A. In any case, I didn’t know about those method interceptors, so in the end, my pair did point me to the right direction. Maybe I should have taken more time to dig through the docs, but this was just a small part of what I needed to do and Stack Overflow was not exactly helpful when trying to find similar behaviour coming from another framework.

The day after the Quarkus implementation, it was time to tackle framework 3: Micronaut. Unfortunately for the developers maintaining it but fortunately for our diverse clientele, our library supports them all. It’s been a while since I dabbled in the JVM ecosystem and I hadn’t heard of it. Makes sense as its first release was in 2018 and my last JVM project before this one was in 2013. Time to hit up our friend again.

This time, most of the proposals didn’t make sense. In one case, ChatGPT suggested to implement a Micronaut interface and got the methods to override completely wrong. Is this the hallucination part? What’s more fascinating, however, is that when you point that out, it again apologizes and corrects itself. So where does it get that second data that does compile (correctness is another thing), and why didn’t it propose that in the first place? This wasn’t a matter of me getting the prompt wrong—it was a very technical and very specific question and there was more than enough context from previous prompts that it clearly was aware of.

I have no idea how these things work under the hood, and I really don’t want to devote my precious free time trying to find out. But it is interesting to see how practical the appliance of these systems can be when you’re on your own, you don’t have any background on a particular framework, want to get stuff done quickly, and can’t find a relevant starting point besides the obvious tutorials that don’t suit your needs. I’ll admit: it was damn handy to get served a few starting points.

However! The creative critical thinking part is yet to be done. Most suggestions are a lot less accurate than old Stack Overflow posts created by humans to help other humans so a quick copy-paste will very quickly lead to failure. Perhaps that was only the case in this particular example where I needed to do something specific and very “framework-heavy”, where other examples the LLM sources its knowledge from are scarcer. There are others out there who try to make fun of ChatGPT’s stumblings by asking it to generate programs. In that case, I would be even more weary of its results. I’d only use it to help explain what you could do with something, not ask it to just come up with the solution…

As Manda Putra said in their How I Feels Pair Programming With AI:

Use it to learn new tools, now that AI is better at reading code type, I think it resulted in a better understanding of how the library that you are using works! I use it a lot to learn some languages like Swift, PHP, Python, and Rust (Although Rust, seems like it suggests the wrong way that’s okay)

My experience hasn’t miraculously converted me into a devotee. I will continue to be a sceptic but it is interesting to see that as a last resort, when no-one else is around (physically or virtually) to get you started and you’re on the clock, ChatGPT et al. can fill in to be your navigator. Just be sure to fact-check everything and never copy something in without very critically evaluating it.

programming   java  ai