How good are LLM agents? Depends on whom you ask!

Depending on whom you ask, you get very different answers. I’ve been noticing a pattern.

If you ask a generalist who is B+/A- on any job you throw at them, they are likely to find LLMs very useful.

If you ask a specialist who has spent a lot of time doing a few things and has deep expertise in them, they are more likely to call LLMs garbage.

I think both of them are right.

My own experience with LLMs is that they are very good at doing “average”. Being a generalist, I only know a few things very well (learnt during my Ph.D. and startup), and I can see LLM sloppiness in those areas. But for most other things, I’ve no way to evaluate LLM output—a plausible answer looks as good as a correct one.

I totally understand when Zig maintainers or database developers ban the use of LLMs. I also understand when a good application developer, who understands some part of her system very well, relies heavily on LLMs for other important jobs like adding a demo site, integration, or a front-end in a language she doesn’t know.

What about enterprise and large organizations? If their products are technically “average” like a social media client, then I see some advantage of using coding agents (though I’m not sure if long-term costs are worth it). If their product has critical components that require very specialized knowledge — such as compilers, database engines, or encryption algorithms — I’d be running away from them if I get to know they are trading their hard-earned tribal knowledge for short-term, LLM-induced gains.

Leave a Reply