Academic research shows that ChatGPT can help developers find errors in their code. The quality of AI-generated code remains a point of discussion.
Software developers like to use tools like Github CoPilot, Codex and also ChatGPT for writing and checking their codes. But how reliable are AI-generated suggestions in software development? Researchers from the universities of London and Mainz put it to the test and let ChatGPT work with forty incorrect codes.
OpenAI’s popular chatbot fared quite well with nineteen bugs fixed. ChatGPT achieved almost the same score as Codex in similar experiments. Codex comes from the same parent company and is also built on the same language model, but is specifically programmed to generate codes where ChatGPT’s capabilities focus on natural interaction.
read also
ChatGPT earns award from American university: is that problematic?
Moreover, ChatGPT has an additional advantage over other tools. The researchers discovered how the chatbot’s communication skills can make a difference in reaching the right solution. ChatGPT can ask for more information about the problem itself and if the chatbot does not immediately find the right solution, you can push it in the right direction with hints. By interacting with the bot, the researchers eventually came up with 31 correct solutions.
AI is help, but not a panacea
The study concludes that ChatGPT and the like can be a useful tool in troubleshooting code errors. In software development it is important to detect errors as early as possible in the process. Do all software developers now have to get started with ChatGPT? That is a matter of debate in the academic world.
Stanford University is less than favorable to a recent study on AI-generated code. In an experiment, software developers who receive the help of artificial intelligence generally delivered less performant and less secure code than colleagues who had to do everything themselves. In the end, therefore, human control is still necessary.
Also, using ChatGPT for code development is not without risks for companies. Amazon has reprimanded software developers for having secret internal code reviewed by ChatGPT. Management fears that the chatbot could reuse that code to solve other users’ problems.