Microsoft continues to lead the way in AI technology, despite the internal struggles at OpenAI. The company has released Orca 2, a pair of small language models that have shown impressive performance in complex reasoning tasks. The 7 billion and 13 billion parameter models were tested against larger models and have achieved remarkable results.

What sets Orca 2 apart is its ability to perform complex reasoning tasks, usually only found in much larger models. Microsoft has open-sourced the models, making them available for further research and development in smaller models that perform as well as their larger counterparts. This is particularly beneficial for enterprises with limited resources, as it offers a more cost-effective solution for addressing targeted use cases without the need for extensive computing capacity.

In addition, Microsoft has made a significant breakthrough in teaching small language models how to reason. Rather than imitating the behavior of larger models, the researchers used tailored synthetic datasets to fine-tune the models to employ different solution strategies for different tasks. The models were trained to determine the most effective solution strategy for each task, providing a more flexible and adaptable approach to reasoning.

The benchmark results of Orca 2 showed that it outperforms larger models in various aspects, including language understanding, common-sense reasoning, math problem solving, and reading comprehension. While the models have inherent limitations, their potential for future advancements, especially in improved reasoning and control, is evident. This marks a significant step in diversifying the applications and deployment options of language models.

As Microsoft continues to lead in the AI space, it’s safe to say that more high-performing small language models are likely to emerge in the near future. Other companies, such as China’s 01.AI and Paris-based startup Mistral AI, are also making strides in this area by developing small, high-performing models that outperform larger offerings. This ongoing research and development in the space of small language models are sure to yield more innovations in the near future.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings. Are you ready to bring more awareness to your brand? Consider becoming a sponsor for The AI Impact Tour. Learn more about the opportunities here.