In my experience, they’re better at poking holes in code than writing it, whether that’s green or brownfield.

I’ve tried to get it to make sections of changes for me, and it feels very productive, but when I time myself I find I spend probably more time correcting the LLM’s work than if I’d just written it myself.

But if you ask it to judge a refactor, then you might actually get one or two good points. You just have to really be careful to double check its assertions if you’re unfamiliar with anything, because it will lead you to some real boners if you just follow it blindly.

But if you find yourself deep in conversation trying to explain to the LLM why it’s not getting your idea, you’re going to wind up with a bad product.

Yes. Kind of. It takes ( a couple of days) experience with LLMs to know that failing to understand your corrections means immediate delete and try another LLM. The only OpenAI llm I tried was their 120g open source release. It insisted that it was correct in its stupidity. That’s worse than LLMs that forget the corrections from 3 prompts ago, though I also learned that is also grounds for delete over any hope for their usefulness.

I’m not above slinging a little spaghetti if it pays the bills.

Nah, it’s the microplastics.

He needs at least a decade of industry experience. That helps me find jobs.

Where is this coming from? I don’t think an LLM can code at the level of a recent cs grad unless it’s piloted by a cs grad.

Maybe you’ve had much better luck than me, but coding LLMs seem largely useless without prior coding knowledge.

What’s this based on? Have you met a fresh CS graduate and compared them to an LLM? Does it not vary person to person? Or fuck it, LLM to LLM? Calling them not qualified seems harsh when it’s based on sod all.

Never, they donate to get the politicians reelected.

This is in no way new. 20 years ago I used to refer to some job postings as H1Bait because they’d have requirements that were physically impossible (like having 5 years experience with a piece of software <2 years old) specifically so they could claim they couldn’t find anyone qualified (because anyone claiming to be qualified was definitely lying) to justify an H1B for which they would be suddenly way less thorough about checking qualifications.

After they fill up on H1B workers and find out that only 1/10 is a good investment.

H1B development work has been a thing for decades, but there’s a reason why there are still high-paying development jobs in the US.

Answer is probably the same as before AI: build a portfolio on GitHub

You really think that using GitHub falls in the usual vibecoding toolbox? As in: would they even know where/how to look?

Working with Accenture

I am so sorry…

a coding LLM can code as well as a fresh CS grad.

For a couple of hundred lines of code, they might even be above average. When you split that into a couple of files or start branching out, they usually start to struggle.

after you give them a piece of advice once or twice, they stop making that same mistake.

That’s a damn good observation. Learning only happens with re-training and that’s wayyy cheaper when done in meat.

Maybe try convincing him in terms he would understand. If it was really that good, it wouldn’t be public. They’d just use it internally to replace every proprietary piece of software in existence. They’d be shitting out their own browser, office suite, CAD, OS, etc. Microsoft would be screwing themselves by making chatgpt public. Microsoft could replace all the Adobe products and drive them out of business tomorrow.

I mean … the first moon landings took a very low number of clicks to make the calculations, technically speaking

it is impossible to convince him that this is, in fact, not the case

He’s probably an investor.

The tech economy is struggling. Every company needs 20% more every year, or it’s considered a failure. The big fish have bought up every promising property on the map in search of this. It’s almost impossible to go from small to large without getting gobbled up, and the guys gobbling up already have 7 different flavors of what you’re trying to make on ice in a repo somewhere. There’s no new venture capital flowing into conventional work.

AI has all the venture capitalists buzzing, handing over money like it’s 1999. Investors are hopping on every hype train because each one has the chance of getting gobbled up and making a good return on investment.

These mega CEO’s have moved their personal portfolios into AI funding and their companies pushing the product will line their pockets indirectly.

At some point, that $200/pp/m price will shoot up. They’re spending billions on datacenters, and eventually those investments will be called in for returns.

When they hit the wall for training-based improvement, things got slippery. Current models are costing exponentially more, making several calls for every request. The market’s not going to bear that without an exponential cost increase, even if they’re getting good work done.

Love it, I have a vibe coding colleague I will use this with.

I like this mentality. I might start telling people the same thing

Lmao glad I could help! I hate those big commits. They’re so much harder to traverse and know what’s going on. Developer experience has been big on my mind lately. Working 5 days a week is already hard, but there are moments when we can make tiny bits easier for each other.

I think storyboards is a great example of how it could be used properly.

Storyboards are a great way for someone to communicate “this is how I want it to look” in a rough way. But, a storyboard will never show up in the final movie (except maybe fun clips during the credits or something). It’s something that helps you on your way, but along the way 100% of it is replaced.

Similarly, the way I think of generative AI is that it’s basically a really good props department.

In the past, if a props / graphics / FX department had to generate some text on a computer screen that looked like someone was Hacking the Planet they’d need to come up with something that looked completely realistic. But, it would either be something hand-crafted, or they’d just go grab some open-source file and spew it out on the screen. What generative AI does is that it digests vast amounts of data to be able to come up with something that looks realistic for the prompt it was given. For something like a hacking scene, an LLM can probably generate something that’s actually much better than what the humans would make given the time and effort required. A hacking scene that a computer security professional would think is realistic is normally way beyond the required scope. But, an LLM can probably do one that is actually plausible for a computer security professional because of what that LLM has been trained on. But, it’s still a prop. If there are any IP addresses or email addresses in the LLM-generated output they may or may not work. And, for a movie prop, it might actually be worse if they do work.

When you’re asking an AI something like “What does a selection sort algorithm look like in Rust?”, what you’re really doing is asking “What does a realistic answer to that question look like?” You’re basically asking for a prop.

Now, some props can be extremely realistic looking. Think of the cockpit of an airplane in a serious aviation drama. The props people will probably either build a very realistic cockpit, or maybe even buy one from a junkyard and fix it up. The prop will be realistic enough that even a pilot will look at it and say that it’s correctly laid out and accurate. Similarly, if you ask an LLM to produce code for you, sometimes it will give you something that is realistic enough that it actually works.

Having said that, fundamentally, there’s a difference between “What is the answer to this question?” and “What would a realistic answer to this question look like?” And that’s the fundamental flaw of LLMs. Answering a question requires understanding the question. Simulating an answer just requires pattern matching.

I think this is great. I like hearing about your experience in the VFX industry since it’s unfamiliar to me as a web dev. The storyboard comparison is spot on. I like that people can drum up a “what if” at such a fast pace, but vibe coders need to be aware that it’s not a final product. You can spin it up, gauge what works and what doesn’t, and now you have feasibility with low overhead. There’s real value to that.

Edit: forgot to touch on your PR comment.

At work, we have an optional GitHub workflow that lets you call Claude in a PR and it will do its own assessment based on the instructions file we wrote for it. We stress that it’s not a final say and will make mistakes, but it’s been good in a pinch. I think if it misses 5 things but uncovers 1 bug, that’s still a win. I’ve definitely had “a-ha” moments with it where my dumb brain failed to properly handle a condition or something. Our company is good about using it responsibly and supplying as much context as we possibly can.

I like your previs analogy, because that’s how I’ve been thinking of it in my head without really knowing how to communicate it. It’s not very good at making a finished project, but it can be useful to demonstrate a direction to go in.

And actually, the one time I’ve felt I was able to use AI successfully was literally using it for previs; I had a specific idea of design I wanted for a logo, but didn’t know how to communicate it. So I created about a hundred AI iterations that eventually got close to what I wanted, handed that to my wife who is an actual artist, told her that was roughly what I was thinking about, and then she took the direction it was going in and made it an actual proper finished design. It saved us probably 15-20 iterations of going back and forth, and kept her from getting progressively more annoyed with me for saying “well… can you make it like that, but more so?”

I will usually google that kind of thing first (to save the rainforests)… Often I can find something that way, otherwise I might try an LLM

Tests are probably both the best and worst things to use LLMs for.

They’re the best because of all the boilerplate. Unit tests tend to have so much of that, setting things up and tearing it down. You want that to be as consistent as possible so that someone looking at it immediately understands what they’re seeing.

OTOH, tests are also where you figure out how to attack your code from multiple angles. You really need to understand your code to think of all the ways it could fail. LLMs don’t understand anything, so I’d never trust one to come up with a good set of things to test.

Unit tests become the specification once they are written. ChatGPT can easily write unit tests from whatever your specification is before that – such as documentation, a bunch of comments and stubs, or even a first draft of the function itself, given enough context from the rest of the project.

Unit tests are too klunky to think in. You don’t prototype the specification by implementing unit tests. And you really only lay down a few critical paths even if you “write the tests first” because code paths always come up during implementation that demand more test coverage anyway.

Also after your 4th proper project you won’t be as confused about “how to get started” anymore anyway.

You have a language of choice, a gui framework, and a build system you are comfortable with in mind already just start making folders.