Back

Bugs

Code review is becoming a real bottleneck in organizations where LLM coding is used. The number of lines that a single developer can produce in a day is higher than the number of lines a developer can feasibly read and understand. Some respond to this fact by saying we don't need to review code anymore. However, I often review code written by high-reasoning LLMs that contains bugs. Those bugs are not easy to catch, because they are not simple localized errors, but rather design decisions that will cause issues in production given the context of the application. On the other hand, I do think LLMs make fewer mistakes than humans, so it does sometimes feel overkill to read every single line of code.

Personally I've been doing code reviews in two passes: a quick first pass to identify which parts of the code need deeper reading, and then a second pass to actually understand and review the code. This reduces the load but I still have the impression that the time spent reviewing is too long.

One could hope that LLM-written tests would preclude code review, which I think is partly true, if one is careful about which tests the LLM introduces. In particular, end-to-end tests are useful. One thing that's been helpful to us is having a reproducible infrastructure that we can spin up in CI, as a faithful replica of production, and then running end-to-end tests against this isolated infra. But unit tests have been fairly useless, and I've been spending some time removing most of the unit tests our LLMs have written over the last few months.

One approach that seems to be growing in popularity is to ship fast and then fix issues when they appear. My perception over the last 6 months is that quality of software has generally dropped. Most new software I've tried has felt buggy, and many services I've used for a long time became less stable. This seems in opposition to the statement that "LLMs produce fewer mistakes," but I think the reason is simply that product companies are shipping more code than before, and as a result shipping more bugs, which increases the perception of instability even if the proportion of bugs per line of code is lower than in human-written code.

Most technical users are now considering switching from existing services to their own home-built ones, because it's cheap and fast to vibe code a substitute that does a good job. If you control the code, then you can fix bugs yourself, probably more quickly than some third-party company. That means that the quality bar for a third-party service is now much higher. So I think it's a bad idea to compromise quality to avoid reviews and quality assurance, and fix bugs when they appear in production, because such software won't pass the minimum bar most people expect. It might have worked when few options existed on the market and product features were hard to build, but that's no longer the reality. I personally churned from many services in the past few months, and the main reason was that the product had too many bugs. This is especially the case for products that can block shipping velocity, for instance if a company offers CI infrastructure (one such case where we had to build our own instead of buying).

Besides testing, typing seems to also help reduce the burden of code reviews, because it makes it easier to understand code and to encode invariants in it, such that there are fewer scenarios one needs to reason about when reading the code. A high-performance language with dependent types would help a lot here, since it could shift the review time more towards checking the specification than the implementation. Apart from it, I don't see many other possible shortcuts, and developing high-quality applications that many people depend on still requires a combination of human judgement (great architecture), end-to-end tests on real-shaped infra, and human code review. I do think this is a transitional state of things and this will soon no longer be true, although timelines are unclear.