A supercomputer in Azure used to train the OpenAI GPT language model

The massive language model of OpenAI, known as GPT, may be trained on a supercomputer created by Microsoft. This project cost several hundred million dollars to complete, and Microsoft now makes its breakthroughs available to both its services and its clients.

As Microsoft and OpenAI began collaborating in 2019, they began developing new computing capabilities for Azure. Large models were intended to be developed in order to speed up the development of artificial intelligence (AI), and the Azure AI supercomputer has in fact trained the language model that powers the GhatGPT chatbot.

Tens of thousands of Nvidia A100 graphics units designed for AI were utilized to build the cloud computing infrastructure. These units were linked together to form “a high-performance, low-latency network” and communicated using Nvidia’s InfiniBand architecture.

Nidhi Chappell, Microsoft’s product manager for Azure High, noted that this scale exceeds what even GPU and network equipment providers have tested. “Uncharted territory was involved. Nobody was certain that the equipment could be stretched that far before breaking, “He continues.

They distributed the processing workload across thousands of GPUs housed in a cluster in order to train a model that large. Chappell says on the Microsoft news page that getting the greatest performance required “a lot of tuning at the system level” in addition to just connecting a bunch of GPUs and putting them to work together.

Due to the departure of Scott Guthrie, Microsoft’s senior vice president in charge of cloud and AI, to Bloomberg, the development of this project has required an investment of several hundred million dollars.

These efforts led Microsoft to launch new virtual machines on Monday that incorporate the most recent Nividia H100 Tensor Core GPUs and the Quantum-2 InfiniBand network, according to Matt Vegas, the principal director of products for Azure HPC + AI.

It concerns the VM ND H100 v5, which is now in preview and supports GPU capacities ranging from eight to thousands. Nevertheless, Microsoft plans to make it “in an Azure standard offering” so that anybody may “unlock the potential of AI at scale in the cloud.”

But, improvements to the OpenAI language model have also been included into other Microsoft services, such as the brand-new Bing, GitHub Copilot, which provides coding advice, or Azure OpenAI Service, for instance. It is anticipated that the latest generation of GPT will be introduced within the next few days.