In a randomized controlled trial, we examined 1) how quickly software developers picked up a new skill (in this case, a Python library) with and without AI assistance; and 2) whether using AI made them less likely to understand the code they’d just written.
We found that using AI assistance led to a statistically significant decrease in mastery. On a quiz that covered concepts they’d used just a few minutes before, participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance.
Who designed this study? I assume it wasn’t a software engineer, because this doesn’t reflect real world “coding skills”. This is just a programming-flavored memory test. Obviously, the people who coded by hand remembered more about the library in the same way students who take notes by hand as opposed to typing tend to remember more.
A proper study would need to evaluate critical thinking and problem solving skills using real world software engineering tasks. Maybe find some already-solved, but obscure bug in an open source project and have them try to solve it in a controlled environment (so they don’t just find the existing solution already).
The study is about the impact AI use has on learning. Their experiment seems to test just that, unlike what you’re describing.
Besides, remembering what you did an hour ago seems like a real world problem to me. Unless one manages to switch project before the bug reports come in
The study is about the impact AI use has on learning. Their experiment seems to test just that, unlike what you’re describing.
The title is literally “How AI assistance impacts the formation of coding skills”. Memorizing APIs isn’t what most people would consiser a “coding skill”.
Debugging, systems design, optimization, research and evaluation, etc are what actually make someone a useful engineer, and are the skills a person develops as they go from junior to senior. Even domain knowledge (like knowing a lot about farming if you’re working on farming software) is more useful than memorizing the API of any framework. The only thing memorization does is it saves you a few minutes from having to read some docs, but that’s minimal impact, and it’s something you pick up normally throughout the course of working on a project anyways. When you finish that project, you might never use that API again, or if you do it might have changed completely when a new version is released.
remembering what you did an hour ago seems like a real world problem to me.
Sure, humans have shitty memory, but that has nothing to do with AI code assistance. There are plenty of non-AI coding assistants that help people with this (like Intellisense/LSP auto complete, which has been around for decades)
Who designed this study? I assume it wasn’t a software engineer, because this doesn’t reflect real world “coding skills”. This is just a programming-flavored memory test. Obviously, the people who coded by hand remembered more about the library in the same way students who take notes by hand as opposed to typing tend to remember more.
A proper study would need to evaluate critical thinking and problem solving skills using real world software engineering tasks. Maybe find some already-solved, but obscure bug in an open source project and have them try to solve it in a controlled environment (so they don’t just find the existing solution already).
The study is about the impact AI use has on learning. Their experiment seems to test just that, unlike what you’re describing.
Besides, remembering what you did an hour ago seems like a real world problem to me. Unless one manages to switch project before the bug reports come in
The title is literally “How AI assistance impacts the formation of coding skills”. Memorizing APIs isn’t what most people would consiser a “coding skill”.
Debugging, systems design, optimization, research and evaluation, etc are what actually make someone a useful engineer, and are the skills a person develops as they go from junior to senior. Even domain knowledge (like knowing a lot about farming if you’re working on farming software) is more useful than memorizing the API of any framework. The only thing memorization does is it saves you a few minutes from having to read some docs, but that’s minimal impact, and it’s something you pick up normally throughout the course of working on a project anyways. When you finish that project, you might never use that API again, or if you do it might have changed completely when a new version is released.
Sure, humans have shitty memory, but that has nothing to do with AI code assistance. There are plenty of non-AI coding assistants that help people with this (like Intellisense/LSP auto complete, which has been around for decades)