The pedestrian technology underneath the hype

We argue that, underneath all the large-language model hype, there is a pedestrian technology that has the potential to automate a class of problems that classical software can never do.

To study the capabilities of large-language models in the context of writing code, we study two extreme cases. First, the application to a critical project with high cognitive complexity, consisting of 40 million lines of code, and second, the application to a simple from-scratch toy project.

For the first study, I tried incorporating the technology into everyday-contributions to LLVM. The suggestions have over a 95% reject-rate, and the good suggestions appear when you're editing one instance of a mechanical pattern, and want to change all instances.

For example, consider the following transformation:

if (match(I, m_c_Xor( m_CombineOr(m_ZextOrTrunc(m_Specific(P1)), m_Specific(P1)), m_CombineOr(m_ZextOrTrunc(m_Specific(P1)), m_Specific(P2)))))

to:

if (match(I, m_c_Xor(m_CastOrSelf(m_Specific(P1)), m_CastOrSelf(m_Specific(P2)))))

This can, in theory, be done by a find-and-replace with a regex, but nobody ever does that for a couple instances.

As another example, consider updating argument names and corresponding comments like:

/// Checks that \p L and \p R are used together in an XOR in the use-def chain /// of \p SI's condition, ignoring any casts. The purpose of this function is to /// ensure that LHSAux from the SimpleRecurrence is used correctly in the CRC /// computation. We cannot check the correctness of casts at this point, and /// rely on the KnownBits propagation to check correctness of the CRC /// computation. static bool isConditionalOnXorOfPHIs(const SelectInst *SI, const PHINode *L, const PHINode *R, const Loop &Lp);

to:

/// Checks that \p P1 and \p P2 are used together in an XOR in the use-def chain /// of \p SI's condition, ignoring any casts. The purpose of this function is to /// ensure that LHSAux from the SimpleRecurrence is used correctly in the CRC /// computation. We cannot check the correctness of casts at this point, and /// rely on the KnownBits propagation to check correctness of the CRC /// computation. static bool isConditionalOnXorOfPHIs(const SelectInst *SI, const PHINode *P1, const PHINode *P2, const Loop &L);

A find-and-replace would match the L in the Loop argument, erroneously replacing it with P1.

You could say that all the automation boils down to a smart and automatic find-and-replace in this study, but it is important to note this class of problems can never be automated by classical software.

---

For the second study, I tried applying the technology to writing a tree-sitter based LLVM IR parser. The entire task is a mechanical chore of reading docs and ample examples in the test suite, and encoding the knowledge in the parser. To study the limitations, I tried generating the entire parser using a prompt, but the result was entirely unusable. When writing the parser from a clean-slate, the suggestions were actually quite good. Consider the auto-complete with the following snippet:

binary_instruction: ($) =>

to:

binary_instruction: ($) => seq($._binary_op, $._type, $.ssa_value, $.ssa_value), _binary_op: () => token( choice( "add", "sub", "mul", "div", "mod", "shl", "shr", "and", "or", "xor", ), ), call_instruction: ($) => seq( optional("tail"), "call", $._type, $.atname, "(", optional($.fn_args), ")", ), compare_instruction: ($) => seq($._compare_op, $._type, $.ssa_value, $.ssa_value), _compare_op: () => token( choice( "eq", "ne", "slt", "sgt", "sle", "sge", "ult", "ugt", "ule", "uge", ), ), cast_instruction: ($) => seq($._cast_op, $._type, $.ssa_value, optional(seq("to", $._type))), _cast_op: () => token( choice( "trunc", "zext", "sext", "fptrunc", "fpext", "fp_to_ui", "fp_to_si", "uitofp", "sitofp", "ptrtoint", "inttoptr", "bitcast", ), ), extract_instruction: ($) => seq(token("extractvalue"), $._type, $.ssa_value, "[", $.ssa_value, "]"), insert_instruction: ($) => seq( token("insertvalue"), $._type, $.ssa_value, $.ssa_value, "[", $.ssa_value, "]", ), load_instruction: ($) => seq(token("load"), $._type, $.ssa_value), phi_instruction: ($) => seq(token("phi"), $._type, $.ssa_value, $.ssa_value), store_instruction: ($) => seq(token("store"), $._type, $.ssa_value, $.ssa_value), switch_instruction: ($) => seq(token("switch"), $._type, $.ssa_value, $.ssa_value), branch_instruction: ($) => seq(token("br"), $._type, $.ssa_value), return_instruction: ($) => seq(token("ret"), $._type, $.ssa_value), unreachable_instruction: ($) => token("unreachable"),

which is actually a smarter version of a copy-paste with some knowledge from the internet. Of course the code needs editing, but this is the kind of copy-pasting you'd normally do before making edits.

The technology was also able to generate 300 tests to exercise the parser, and the tests needed little tweaking to pass: this task can be boiled down to copying reduced-case LLVM IR from the test-suite, and matching them with parser nodes. Of course the generated tests aren't high-quality, but about half the tests are usable as-is, and a lot of tedium has been eliminated.

To conclude, the technology is at alpha-stage, as the automation comes at the cost of putting up with bad visual feedback and fighting the auto-complete nearly all the time, but promises a new class of mechanical automation, whose utility is higher in small codebases following mechanical patterns.