GitHub Copilot review

 Although the Copilot technical preview does not always provide nice, correct, or even running code, it is nevertheless useful. Future versions may prove to be significant time saves.

GitHub Copilot, dubbed "Your AI pair programmer" and currently in a limited technical preview, aims to help automate programming in a way that goes beyond what IntelliSense and similar tools can do. It isn't totally self-contained. As we'll see, you must specify (write out) your objectives before Copilot can produce useful code, and you must also supervise Copilot to keep it on track when it inevitably goes off the tracks.

Copilot is a cloud service that connects to Visual Studio Code (on your local machine or in the cloud via GitHub Codespaces), JetBrains IDEs like IntelliJ IDEA, and Neovim. OpenAI Codex, a language model trained on billions of lines of public code, powers the cloud service, which is a code prediction engine.

Yes, there was a debate over Codex and Copilot. Before you get all worked up about Copilot's possible copyright and privacy infringement (I'm looking at you, Free Software Foundation), keep in mind that Codex was trained on publicly available code in a way that the machine learning community considers to be fair use.

You should also keep in mind that Codex is a code synthesizer rather than a search engine. The Copilot creators admit that this isn't the final word on the subject:

… this is a new space, and we are keen to engage in a discussion with developers on these topics and lead the industry in setting appropriate standards for training AI models.

What is GitHub Copilot and how does it work?

"OpenAI Codex was trained on publicly available source code and natural language, so it understands both programming and human languages," according to the GitHub page. The GitHub Copilot editor extension sends your comments and code to the GitHub Copilot service, which subsequently synthesizes and suggests individual lines and entire functions using OpenAI Codex." Furthermore, the service makes advantage of user preferences to optimize future suggestions.

GitHub Copilot is a service that leverages the OpenAI Codex language model to make suggestions based on the content of Visual Studio Code and a few other editors, as seen in this figure.

Visual Studio Code is being used to test GitHub Copilot.

Copilot is currently at a limited technical preview stage. You must first apply for a spot on the preview program waitlist before you can install it.

After you've received your welcome e-mail, go to the Visual Studio Code Marketplace's GitHub Copilot extension page and install the extension. The extension must then be authorized in Visual Studio Code. You can do a tutorial starting at point number 2 on the getting started page. In this article, you'll learn how to make a.JS file, as well as how to use it.

function calculateDaysBetweenDates(begin, end) {

 and you'll end up with a fully functional function based on the function name. Even though the function name is too vague to be useful, you input a comment summarizing what a function should perform in the following tutorial on this page, and you end up with a fully constructed function inferred from the comment.

The GitHub Copilot extension page in the Visual Studio Code marketplace. As you can see at the top, I have already installed the extension.

  

A screenshot of Visual Studio Code with the GitHub Copilot active. I have just started the first tutorial in the documentation, and you can see the “ghost” code suggestion below what I typed, as well as the pop-up Copilot control bar. I created the new file as TypeScript rather than JavaScript, mostly because I’m bloody-minded. In this case Copilot generated JavaScript code anyway.

Copilot capabilities on GitHub

Copilot can deduce function bodies from other code in the file you're working, as well as variable names, in addition to the function name and a summary comment. In TypeScript, for example, if I type a colon after a variable name, Copilot will try to fill in the type. Copilot will cue on the word "test" and produce a runnable test for the previous function if I type "var test1 =". Copilot will try to produce more examples of the same pattern if I input many lines that form a recurring pattern.

Copilot is capable of working with a wide range of frameworks and languages. Python, JavaScript, TypeScript, Ruby, Go, and, more recently, Java are the finest options, with the C family of languages (C, C++, and C#) on the horizon. Others have told me that it works nicely with popular JavaScript frameworks like

Lines 8 and 9 were created by typing the line's beginnings and a number of tabs. Lines 10 and 11 were typed by me, while line 11 was completed by Copilot. As you can see at the bottom of the screen, I was working in TypeScript after compiling it to JavaScript, which I then ran in Node.js. Lines 8 and 9 include inaccurate generated comments about the expected result values.

To begin with, Copilot does not always provide decent code. It doesn't necessarily produce good code. Worse, it doesn't always produce executable code. (During my testing, I came across all three scenarios.)

You must thoroughly examine the code that Copilot generates. Treat it as if it were authored by a green programming intern who knows how to use Google but requires constant supervision.

Use the "Open Copilot" option on the context menu, or the Ctrl-Enter key combination, to open the Copilot recommendations window in a different tab, to avoid accepting the first sample that Copilot gives. Examine all ten suggestions and choose the one that comes closest to your true desires. After that, you may wish to tweak the generated code to make it more robust.

Copilot code creation was tested by GitHub:

We recently ran tests against a group of Python routines with high test coverage in open source repositories. The function bodies were left blank and GitHub Copilot was tasked with filling them up. When given 10 trials, the model got it right 43% of the time on the first try and 57% of the time when given 10 attempts. It's also getting smarter by the day.

Even though it's an impressive achievement for a novel code generation technology, 43 percent correct isn't a very good (or even acceptable) accuracy score for production use. However, if you're a skilled code reviewer, you can alter Copilot-generated code to make it more correct and robust much faster than you could if you were writing it from scratch, especially if you're working with a new library or framework.

Overall, in its current technical preview stage of development, GitHub Copilot is somewhat beneficial. Its current performance makes me optimistic that it will continue to save me time in the future. If and when the anticipated commercial Copilot product is available, whether it will be worthwhile to purchase it is an open question that will depend not just on its improved performance but also on your own abilities and role.

There are a number of products that claim to be competitors to GitHub Copilot. Tabnine, from a Tel Aviv-based company of the same name, appears to be the most promising of these. Tabnine resembles IntelliSense on steroids, and it can train on both your own code corpus and open source code. A number of the other options just scan StackOverflow for similar code, which makes me suspicious of their approach.

It's definitely worth trying out GitHub Copilot in your own setting and keeping track of its growth.