BenchLLM

BenchLLM: The Ultimate LLM Evaluation Tool

What is BenchLLM? BenchLLM is an open-source framework designed to evaluate and test applications powered by Large Language Models (LLMs). It allows AI engineers to build test suites, generate quality reports, and monitor model performance. It supports automated, interactive, and custom evaluation strategies, providing flexibility and power without compromising on predictable results.

Key Features:

• Flexible API: BenchLLM supports OpenAI, Langchain, and any other API out of the box.
• Powerful CLI: Run and evaluate models with simple CLI commands, ideal for CI/CD pipelines.
• Easy Evaluation: Define tests intuitively in JSON or YAML format.
• Organized Tests: Easily organize tests into versionable suites.
• Automation: Automate evaluations in CI/CD pipelines.
• Reporting: Generate and share evaluation reports.
• Performance Monitoring: Detect regressions in production by monitoring model performance.

How does BenchLLM work?

BenchLLM enables AI engineers to evaluate their code and LLMs effectively through several steps:

1. Instantiate Test Objects: Define tests by creating Test objects with inputs and expected outputs.
2. Generate Predictions: Use a Tester object to run the tests and generate predictions from your model.
3. Evaluate Models: Employ an Evaluator object, such as SemanticEvaluator, to evaluate the model's predictions.

Here’s a basic example:

from benchllm import SemanticEvaluator, Test, Tester
from langchain.agents import AgentType, initialize_agent
from langchain.llms import OpenAI

## Keep your code organized in the way you like
def run_agent(input: str):
    llm=OpenAI(temperature=0)
    agent = initialize_agent(
        load_tools(["serpapi", "llm-math"], llm=llm),
        llm=llm,
        agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION
    )
    return agent(input)["output"]

## Instantiate your Test objects
tests = [
    Test(
        input="When was V7 founded? Divide it by 2",
        expected=["1009", "That would be 2018 / 2 = 1009"]
    )
]

## Use a Tester object to generate predictions
tester = Tester(run_agent)
tester.add_tests(tests)
predictions = tester.run()

## Use an Evaluator object to evaluate your model
evaluator = SemanticEvaluator(model="gpt-3")
evaluator.load(predictions)
evaluator.run()

Powerful CLI for CI/CD Integration

BenchLLM features a powerful Command Line Interface (CLI) that enables seamless integration into CI/CD pipelines. You can run tests and evaluate models using simple CLI commands, making it easier to monitor model performance and detect regressions in production.

Flexible API for Custom Evaluations

BenchLLM's flexible API supports OpenAI, Langchain, and virtually any other API. This allows you to test your code on the fly and use multiple evaluation strategies, providing insightful reports tailored to your specific needs.

How to use BenchLLM?

To get started with BenchLLM, follow these steps:

1. Download and Install: Download and install BenchLLM.
2. Define Tests: Define your tests in JSON or YAML format.
3. Run Tests: Use the CLI or API to run your tests.
4. Generate Reports: Generate evaluation reports and share them with your team.

Here’s an example of how to define a test using the @benchllm.test decorator:

import benchllm
from benchllm.input_types import ChatInput
import openai

def chat(messages: ChatInput):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=messages
    )
    return response.choices[0].message.content.strip()

@benchllm.test(suite=".")
def run(input: ChatInput):
    return chat(input)

Who is BenchLLM for?

BenchLLM is ideal for:

• AI Engineers who want to ensure the quality and reliability of their LLM-powered applications.
• Developers looking for a flexible and powerful tool to evaluate their models.
• Teams that need to monitor model performance and detect regressions in production.

Why choose BenchLLM?

• Open-Source: Benefit from a transparent and community-driven tool.
• Flexibility: Supports various APIs and evaluation strategies.
• Integration: Seamlessly integrates into CI/CD pipelines.
• Comprehensive Reporting: Provides insightful reports to track model performance.

BenchLLM is built and maintained with ♥ by V7, a team of AI engineers passionate about building AI products. The tool aims to bridge the gap between the power and flexibility of AI and the need for predictable results.

Share your feedback, ideas, and contributions with Simon Edwardsson or Andrea Azzini to help improve BenchLLM and make it the best LLM evaluation tool for AI engineers.

By choosing BenchLLM, you ensure that your LLM applications meet the highest standards of quality and reliability. Download BenchLLM today and start evaluating your models with confidence!