A Comparison of the Effectiveness of ChatGPT and Co-Pilot for Generating Quality Python Code Solutions

Abstract

Artificial intelligence (AI) has become increasingly popular in software development to automate tasks and improve efficiency. AI has the potential to help while developing or maintaining software, in the sense that it can produce solutions out of a textual requirement specification, and understand code to provide suggestion on how a new requirement could be implemented. In this paper, we focus on the first scenario. Two AI-powered tools that have the potential to revolutionize the way software is developed are OpenAI’s ChatGPT and GitHub’s Copilot. In this paper, we used LeetCode, a popular platform for technical interview preparation and personal upskilling (self-learning), to evaluate the effectiveness of ChatGPT and Copilot on a set of coding problems, along with ChatGPT’s ability to correct itself when provided with feedback. The analysis of the effectiveness can lead to various conclusions, such as on if these solutions are ready to take over coding roles, and to what extent several parameters (difficulty and quality requirements) influence this result. Solutions have been generated for 60 problems using ChatGPT and Copilot, for the Python programming language. We investigated the performance of the models, the recurrent kinds of errors, and the resulting code quality. The evaluation revealed that ChatGPT and Copilot can be effective tools for generating code solutions for easy problems while both models are prone to syntax and semantic errors. Small improvements are observed for ode quality metrics across iterations, although the improvement pattern is not consistently monotonic, questioning ChatGPT’s awareness of the quality of its own solutions. Nevertheless, the improvement that was found along iterations, highlights the potential of AI and humans, acting as partners, in providing the optimal combination. The two models demonstrate a limited capacity for understanding context. Although AI-powered coding tools driven by large language models have the potential to assist developers in their coding tasks, they should be used with caution and in conjunction with human coding expertise. Developer intervention is necessary not only to debug errors but also to ensure high-quality and optimized code.