Vibe Coding Python: Testing Copilot vs. CodeGPT vs. Tabnine
I put three vibe coding tools to the test. Not with the goal of finding the best one, just to see how they’re different. Is one easier? I used three free tools so you can try this project as well and see what your AI comes up with. The tools I chose were GitHub Copilot (obviously), CodeGPT and Tabnine. I wrote my code in VS Code.
The app I chose to build was a simple calculator app that you can interact with in the terminal. I did want to go a little bigger at first, and try a tip calculator app complete with HTML and CSS, but after about 10 minutes, I realized I could complete this experience with something simpler.
Here are the prompts I came up with as a baseline:
Here’s what I discovered along the way.
GitHub Copilot
I started with GitHub Copilot. GitHub Copilot is a chatbot. I clicked the file I wanted to work with, and that is how GitHub Copilot knew which language to code in. With GitHub Copilot, I copied/pasted all prompts into the chat at the same time to see what it would do.
It returned the full application, reading each prompt. When I ran the file in my terminal, I had a syntax error. I copied/pasted the full function (including buggy and non-buggy code) back into GitHub Copilot, and it was able to fix the bug immediately. I updated the code file. After about 1.5 iterations, the calculator was fully functional.
Here’s the code it provided:
CodeGPT
Next, I explored CodeGPT. I accessed it via the command palette (Cmd + Shift + P) and selected CodeGPT: Reset to start fresh. One thing I immediately noticed is that CodeGPT doesn’t handle bulk prompts. You can only enter one at a time.
After it provided code for a prompt, it suggested the next steps. For example, after I asked it to enter the first number, it immediately prompted for the second number. I decided to follow the CodeGPT prompts all the way to the end.
The CodeGPT prompts included error handling, and I was able to interact with the calculator in the terminal on the first try. One thing I did notice, though, is that the CodeGPT prompted app terminated after only one calculation.
Here’s the code:
I did another one using CodeGPT, this time copying and pasting the prompts as I originally wrote them. This is what CodeGPT came up with:
Tabnine
The last tool I used for this experiment is Tabnine. The first go around, I followed the prompts it offered after I gave the first one. They looked oddly similar to the prompts CodeGPT provided. Full disclosure, this is my first time vibe coding, I wasn’t sure if the prompts were saved somehow or if Tabnine and CodeGPT came up with similar prompts.
One thing that was different with Tabnine, though, is that the prompting didn’t end. If I accepted a prompt, it offered code. The last few prompts were the same. It kept asking if the user wanted to calculate again. I did decide not to follow the last prompt and try to run the app in my terminal.
I was unable to do so due to an error. I ran through this pretty quickly but I didn’t see any easy bug fix options like I did with GitHub Copilot. Ultimately decided to try Tabnine again and this time follow the prompts on my list.
Here’s the code using Tabnine’s prompts:
Here’s the code using my prompts:
Takeaways:
Vibe coding can be easier, but you really need to know what you’re doing. This was a simple calculator app, but it gets much more complicated. You need to build in some error handling and think through how you want the application to run. If I had to choose my favorite, and I really only worked in these technologies for about an hour each, I’d say GitHub Copilot. I preferred the chatbot over the inline code. I liked how it fixed the small bug quickly.
