OpenAI today unveiled its new o1 series of AI models, designed with stronger logical reasoning capabilities than previous models. One of the first applications being explored for these new models is improving GitHub Copilot"s ability to analyze, refine, and optimize code.
OpenAI and GitHub have been running some internal tests of the integration of the new o1-preview model with Copilot. One such test was to use o1-preview on performance optimization for one of the most critical code components in Copilot. As noted in the blog, it deeply analyzed the code, understood the constraints and edge cases, and suggested a far better solution in one try.
Another example pitted o1-preview against the current model, GPT-4o, in debugging a browser performance issue within GitHub"s codebase. The bug was causing a focus management function to slow down, stalling the browser.
o1-preview was able to identify the issue and suggest line-by-line fixes in minutes, while GPT-4o gave less targeted responses.
The results highlight how o1-preview’s reasoning capability allows a deeper understanding of the code’s constraints and edge cases, which helps produce a more efficient and higher quality result. Meanwhile, GPT-4o sticks to obvious optimizations and would need a developer’s help to steer Copilot towards more complex approaches.
According to the blog, the o1-preview model has enhanced reasoning which will improve Copilot"s support for code review, refactoring and optimization use cases that rely on richer analyses. It calculates metrics and benchmarks directly from code output and presents results in a concise manner.
With 1,000 elements managed by this code, it was hard to isolate the problem. Eventually we implemented a change that improved the runtime of this function from over 1,000ms to about 16ms. If we had Copilot with o1-preview, we could have quickly identified the problem and fixed it faster.
OpenAI and GitHub plan to offer o1-preview and a smaller o1-mini model on the GitHub Marketplace. While o1-preview remains in preview, developers can apply for early access via Azure AI to experiment with building projects using the new models.