S&P Global recently launched S&P AI Benchmarks by Kensho, which is a benchmarking solution to assess and rank the quantitative reasoning abilities and expertise of large language models (LLMs) across the business and finance industry.

The company said results of these assessments will be displayed on a scoreboard to provide a “transparent and comprehensive” view of the performance of different LLMs.

S&P Global sees S&P AI Benchmarks by Kensho as potentially overhauling the way financial professionals evaluate which LLM to leverage for their financial workflows. The solution can measure and rank how well LLMs can solve complex quantitative reasoning questions, understand business fundamentals and extract relevant financial information from documents.

By rigorously validating the model’s outputs, executives said S&P AI Benchmarks by Kensho can offer accurate and reliable assessments of LLM applications for financial professionals.

“We’re delivering what started as an in-house tool to help enhance our technology into our customers’ hands,” said Bhavesh Dayalji, chief AI officer for S&P Global and CEO of Kensho. “It serves as further evidence of S&P Global’s longstanding commitment to harnessing AI in ways that consistently enhance our data and experiences for our customers.”

S&P Global highlighted that what makes S&P AI Benchmarks by Kensho stand out is its validation process, developed in partnership with a diverse team of specialists. This includes esteemed academics, seasoned researchers, subject matter experts across various domains, and financial professionals from across S&P Global’s divisions, all contributing to the tool’s precision and efficacy.

One of the key advantages of S&P AI Benchmarks by Kensho is its secure submission process. Financial professionals can benchmark their LLMs without sharing any part of their model, only its outputs.

S&P Global said it has been at the forefront of AI for the financial services industry since acquiring Kensho in 2018. Since then, the teams have worked closely together to deliver solutions for customers using Kensho’s core capabilities of machine learning, natural language processing, and data discovery.

S&P Global aims to redefine the intelligence that powers global markets with AI, embracing the transformative potential of GenAI across the enterprise to drive innovation, speed up time to market, and build sophisticated, intuitive experiences for customers.

“As GenAI and LLMs become pervasive across products and services, it is critical that we keep accuracy and reliability at the forefront of what we do,” Dayalji said. “This solution is a huge step forward in ensuring that we are building solutions our customers can rely on.”

For more information about S&P AI Benchmarks by Kensho, visit benchmarks.kensho.com.