OpenRouter Launches Ori Eval to Automate AI Model Selection for Developers

Ori Eval automates model selection by scanning codebases and running real-task benchmarks, prioritizing cost, speed, or performance while integrating with GitHub Actions for regression testing.

Woofun AI reports that OpenRouter has launched Ori Eval, a tool designed to assist developers in selecting optimal AI models for their applications. The system scans codebases to locate model calls and executes multiple candidate models against actual project tasks, allowing users to prioritize performance, speed, or cost. During evaluation, Ori Eval locks testing frameworks, configurations, and inference intensity to ensure comparable results, providing direct rankings rather than relying on general capability metrics.

Beyond answer quality, the tool verifies correct tool usage by agents and prevents unnecessary operations. It can generate tests replicating bugs described in natural language and integrates with GitHub Actions to automatically check for issue recurrence after fixes. This transforms manual trial-and-error into an automated workflow based on real-world business scenarios, addressing the limitations of public rankings for specific use cases like customer service or programming.