Alexander Junge's website

I have used GitHub Copilot, an ML pair programming tool based on OpenAI Codex, for a few months now when working on hobby projects and wanted to share a short anecdote here illustrating its usefulness and where it currently fails.

In general, I believe that Copilot saves time while coding and I primarily use it as a “tab completion on steroids”. The core problem is, as with all text output generated by large language models, that Copilot has no notion of certainty and will always produce an answer. This severely limits its usefulness because I always need to check its output in detail, keeping me so far from using Copilot for generating larger code blocks, such as generating whole functions from docstrings.

An example: Copilot generating comments for PyTorch code

Recently, I came across a fun example that illustrates this behavior quite well. I wrote a PyTorch training loop and noticed that Copilot started generating useful inline comments:

# Training loop
for _ in tqdm.tqdm(range(num_epochs)):
    for data_inputs, data_labels in data_loader:

        # 1: move to device
        data_inputs = data_inputs.to(device)
        data_labels = data_labels.to(device)

        # 2: forward pass
        preds = model(data_inputs)

        # 3: compute loss
        loss = loss_module(preds, data_labels)

        # 4: compute gradients
        optimizer.zero_grad()
        loss.backward()

        # 5: update parameters
        optimizer.step()

        # 6: compute accuracy

The last line was generated by Copilot after I typed “#” followed by a tab. And so were the following lines:

        # 7: log

        # 8: save model

        # 9: save optimizer

        # 10: save metrics

        # 11: save predictions

        # 12: save labels

        # 13: save loss

        # 14: save gradients

        # 15: save inputs

        # 15: save outputs

        # 16: save activations

        # 17: save weights

        # 18: save biases

        # 19: save histograms

While the above makes sense, after some time it would just generate non-sensical output without ever stopping:

        # 20: save images

        # 21: save videos

        # 22: save audio

        # 23: save text

        # 24: save plots

        # 25: save figures

        # 26: save tables

        # 27: save dataframes

        # 28: save tensors

        # 29: save numpy arrays

        # 30: save scipy arrays

        # 31: save pandas arrays

        # 32: save matplotlib arrays

        # 33: save seaborn arrays

        # 34: save bokeh arrays

        # 35: save plotly arrays

        # 36: save altair arrays

        # 37: save networkx arrays

        # 38: save pytorch arrays

        # 39: save tensorflow arrays

        # 40: save keras arrays

        # 41: save sklearn arrays

        # 42: save xgboost arrays

        # 43: save lightgbm arrays

        # 44: save catboost arrays

        # 45: save dask arrays

        # 46: save ray arrays

        # 47: save jax arrays

        # 48: save gluon arrays

        # 49: save mxnet arrays

        # 50: save chainer arrays

        # 51: save onnx arrays

        # 52: save caffe arrays

        # 53: save caffe2 arrays

        # 54: save tensorflow lite arrays

        # 55: save tensorflow js arrays

        # 56: save pytorch js arrays

        # 57: save pytorch mobile arrays

        # 58: save pytorch c++ arrays

        # 59: save pytorch c arrays

        # 60: save pytorch cuda arrays

        # 61: save pytorch cuda c arrays

        # 62: save pytorch cuda c++ arrays

        # 63: save pytorch cuda js arrays

        # 64: save pytorch cuda mobile arrays

        # 65: save pytorch cuda lite arrays

        # 66: save pytorch cuda onnx arrays

        # 67: save pytorch cuda caffe arrays

        # 68: save pytorch cuda caffe2 arrays

        # 69: save pytorch cuda tensorflow arrays

        # 70: save pytorch cuda keras arrays

        # 71: save pytorch cuda sklearn arrays

        # 72: save pytorch cuda xgboost arrays

        # 73: save pytorch cuda lightgbm arrays

        # 74: save pytorch cuda catboost arrays

        # 75: save pytorch cuda dask arrays

        # 76: save pytorch cuda ray arrays

        # 77: save pytorch cuda jax arrays

        # 78: save pytorch cuda gluon arrays

        # 79: save pytorch cuda mxnet arrays

        # 80: save pytorch cuda chainer arrays

        # 81: save pytorch cuda onnx arrays

        # 82: save pytorch cuda caffe arrays

        # 83: save pytorch cuda caffe2 arrays

        # 84: save pytorch cuda tensorflow lite arrays

        # 85: save pytorch cuda tensorflow js arrays

        # 86: save pytorch cuda pytorch js arrays

        # 87: save pytorch cuda pytorch mobile arrays

        # 88: save pytorch cuda pytorch c++ arrays

        # 89: save pytorch cuda pytorch c arrays

        # 90: save pytorch cuda pytorch js arrays

        # 91: save pytorch cuda pytorch mobile arrays

        # 92: save pytorch cuda pytorch lite arrays

        # 93: save pytorch cuda pytorch onnx arrays

        # 94: save pytorch cuda pytorch caffe arrays

        # 95: save pytorch cuda pytorch caffe2 arrays

        # 96: save pytorch cuda pytorch tensorflow arrays

        # 97: save pytorch cuda pytorch keras arrays

        # 98: save pytorch cuda pytorch sklearn arrays

        # 99: save pytorch cuda pytorch xgboost arrays

        # 100: save pytorch cuda pytorch lightgbm arrays

        # 101: save pytorch cuda pytorch catboost arrays

        # 102: save pytorch cuda pytorch dask arrays

This is where I gave up but the model happily droned on.

Summary

Of course, this is only one example and the debate around the use of tools like Copilot is much more nuanced, including copyright discussions around the training data. I nevertheless think that this example nicely shows some highlights and lowlights of ML pair programming tools. Right now, Copilot feels more like a pair programmer that is hard to work with in the sense that it always confidently answers without helping you to decide if the answer is right or wrong.

As this is the last day of 2022, I wish everybody reading this a happy new year and a great start into 2023