Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
yes they can. I regularly do. Regexes aren’t hard to write, their logic is quite simple. They’re hard to read, yes, but they are almost always one-offs (ex, substitutions in nvim).
yes, “complex” regexes are quite simple too. Complex regexes are long, not difficult. They appear complex because you have to “inline” everything. They really are not that hard.
cryptic != complex. Are they cryptic? yes. Are they complex? not really, if you can understand “one or more” or “zero or more” and some other really simple concepts like “one of these” or “not one of these” or “this is optional”. You could explain these to a child. It’s only because they look cryptic that people think they are complex. Unless you start using backreferences and advanced concepts like those (which are not usually needed in most cases) they are very simple. long != complex
tests can never prove correctness of code. All they can prove is “the thing hasn’t failed yet”. Proper reasoning is always needed if you want a guarantee.
If you had the llm write the regex for you, I can practically guarantee that you won’t think of, and write tests for, all the edge cases.
No, which is why I avoid regexes for most production code and also why I would never use one written by a pathological liar and always guessing coder like an LLM.
LLM is great when you’re coding in a pure fictional programming language like elm and are using loss of custom types to make impossible states unrepresentable, and the function you’re writing could have been derived by the Haskell compiler, so mathematically the only possible way you could write it wrong is to use the wrong constructor, then it’s usually right and when it’s wrong either it doesn’t compile or you can see it’s chosen the wrong path.
The rest of the time it will make shit up and when you challenge it, out will happily rewrite it for you, but there’s no particular reason why it wouldn’t make up more nonsense.
Regexes are far easier to write than to debug, which is exactly why they’re poison for a maintainable code base and a really bad use case for an LLM.
I also wouldn’t use an LLM for languages in which there are lots and lots of ways to go wrong. That’s exactly when you need an experienced developer, not someone who guesses based on what they read online and no understanding, never learning anything, because, my young padawan, that’s exactly what an LLM is, every day.
If you find yourself writing regexes often enough that speeding up that process would increase your productivity by “a lot”, then you should get good at writing them yourself which means practicing without an LLM. If its something that you don’t do often enough to warrant getting good at then the productivity increase is negligible.
I think the main benefit here isn’t really productivity but developer comfort by saving them from having to step out of their comfort zone.
does the regex search for what you wanted to? Does it work in all cases? Can I be confident that it will find all instances i care about, or will I still have to comb the code manually?
Depends on what you need to match. Regex is just another programming language. It’s more declarative than traditional languages though (it’s basically pattern matching).
Pattern matching is something I already do a lot of in my code, so regexes aren’t that much different.
Regardless, the syntax sucks. It takes some time to get familiar with it, but once you get past that, it’s really simple.
LLMs making you code faster means your slow not LLMs fast
I doubt anyone can write complex regex in ~30 seconds, LLM’s can
yes they can. I regularly do. Regexes aren’t hard to write, their logic is quite simple. They’re hard to read, yes, but they are almost always one-offs (ex, substitutions in nvim).
He did say complex regex. A complex regex is not simple.
yes, “complex” regexes are quite simple too. Complex regexes are long, not difficult. They appear complex because you have to “inline” everything. They really are not that hard.
This is stupid pedantry. By that logic literally nothing is complex because everything is made up of simple parts.
cryptic != complex. Are they cryptic? yes. Are they complex? not really, if you can understand “one or more” or “zero or more” and some other really simple concepts like “one of these” or “not one of these” or “this is optional”. You could explain these to a child. It’s only because they look cryptic that people think they are complex. Unless you start using backreferences and advanced concepts like those (which are not usually needed in most cases) they are very simple. long != complex
Ok I can see you haven’t actually come across any complex regexes yet…
(Which is probably a good thing tbh - if you’re writing complex regexes you’re doing it wrong.)
i haven’t come across many. But i have written a lot.
I’m not trusting a regex written by AI
You don’t have to. You can read it.
That’s why you write tests
tests can never prove correctness of code. All they can prove is “the thing hasn’t failed yet”. Proper reasoning is always needed if you want a guarantee.
If you had the llm write the regex for you, I can practically guarantee that you won’t think of, and write tests for, all the edge cases.
You formally verify your regexes? Doubtful.
No, which is why I avoid regexes for most production code and also why I would never use one written by a pathological liar and always guessing coder like an LLM.
LLM is great when you’re coding in a pure fictional programming language like elm and are using loss of custom types to make impossible states unrepresentable, and the function you’re writing could have been derived by the Haskell compiler, so mathematically the only possible way you could write it wrong is to use the wrong constructor, then it’s usually right and when it’s wrong either it doesn’t compile or you can see it’s chosen the wrong path.
The rest of the time it will make shit up and when you challenge it, out will happily rewrite it for you, but there’s no particular reason why it wouldn’t make up more nonsense.
Regexes are far easier to write than to debug, which is exactly why they’re poison for a maintainable code base and a really bad use case for an LLM.
I also wouldn’t use an LLM for languages in which there are lots and lots of ways to go wrong. That’s exactly when you need an experienced developer, not someone who guesses based on what they read online and no understanding, never learning anything, because, my young padawan, that’s exactly what an LLM is, every day.
Watch your LLM like a hawk.
You have definitely never worked with a regex guru.
No, but not everyone is a regex guru.
If AI can write code half as good and fast as a regex guru, it’s going to increase the average dev’s productivity a lot
If you find yourself writing regexes often enough that speeding up that process would increase your productivity by “a lot”, then you should get good at writing them yourself which means practicing without an LLM. If its something that you don’t do often enough to warrant getting good at then the productivity increase is negligible.
I think the main benefit here isn’t really productivity but developer comfort by saving them from having to step out of their comfort zone.
does the regex search for what you wanted to? Does it work in all cases? Can I be confident that it will find all instances i care about, or will I still have to comb the code manually?
Depends on what you need to match. Regex is just another programming language. It’s more declarative than traditional languages though (it’s basically pattern matching).
Pattern matching is something I already do a lot of in my code, so regexes aren’t that much different.
Regardless, the syntax sucks. It takes some time to get familiar with it, but once you get past that, it’s really simple.