Getting Started

​

Hebo Eval CLI

​

Installation

bun install -g hebo-eval@latest

​

Basic Usage

hebo-eval run <agent> [options]

​

Command Options

​

Configuration

1. Environment Variables:
   Copy

   export HEBO_API_KEY=your_api_key_here
   export OPENAI_API_KEY=your_openai_key_here
   export HEBO_EMBEDDING_API_KEY=your_embedding_key_here
   
2. Configuration File: Create a YAML configuration file (e.g., hebo-evals.config.yaml):
   Copy

   # Hebo Eval Configuration Template
   # Copy this file to hebo-evals.config.yaml and replace the values with your own
   
   # Provider configurations
   providers:
     openai:
       provider: openai
       baseUrl: https://api.openai.com/v1
       apiKey: ${OPENAI_API_KEY} # Will be replaced with the value of OPENAI_API_KEY environment variable
       authHeader:
         name: Authorization
         format: Bearer ${OPENAI_API_KEY} # Can use the same environment variable multiple times
   
     hebo:
       provider: hebo
       baseUrl: https://app.hebo.ai
       apiKey: ${HEBO_API_KEY} # Will be replaced with the value of HEBO_API_KEY environment variable
       authHeader:
         name: Authorization
         format: Bearer ${HEBO_API_KEY}
   
   # Default provider to use if not specified in the command
   defaultProvider: hebo
   
   # Embedding configuration
   embedding:
     provider: hebo
     model: hebo-embeddings
     baseUrl: https://api.hebo.ai/v1
     apiKey: ${HEBO_EMBEDDING_API_KEY} # Can reuse the same environment variable
   
   Note: The configuration file supports environment variable substitution using ${VARIABLE_NAME} syntax. This allows you to keep sensitive information like API keys in environment variables while referencing them in your configuration file.

​

Configuration Template

• Multiple Provider Support: Configure both OpenAI and Hebo providers
• Environment Variable Integration: Use ${VARIABLE_NAME} syntax for secure API key management
• Flexible Authentication: Support for different authentication header formats
• Embedding Configuration: Separate configuration for embedding models

1. Copy the template to your project directory:
   Copy

   cp hebo-evals.config.yaml ./your-project/
   
2. Set up your environment variables:
   Copy

   export OPENAI_API_KEY=your_openai_key_here
   export HEBO_API_KEY=your_hebo_key_here
   export HEBO_EMBEDDING_API_KEY=your_embedding_key_here
   
3. Customize the configuration as needed for your specific use case.

​

Provider Mapping Logic

• OpenAI: Any model name that starts with gpt- (e.g., gpt-3.5-turbo, gpt-4) is mapped to the OpenAI provider.
• Hebo: Any model name that contains a colon (:) (e.g., gato-qa:v1) is mapped to the Hebo provider.
• Anthropic: Any model name that starts with claude- (e.g., claude-2, claude-instant) is mapped to the Anthropic provider.
• Custom: Any model name that starts with custom- is mapped to the Custom provider.

hebo-eval run gpt-4         # Uses OpenAI provider
hebo-eval run gato-qa:v1    # Uses Hebo provider
hebo-eval run claude-2      # Uses Anthropic provider (future support)
hebo-eval run custom-foo    # Uses Custom provider (future support)

Note:
Currently, Hebo Eval only supports OpenAI and Hebo as providers. Support for additional providers such as Anthropic and Custom is planned for future releases.

​

Test Cases

​

Test Case Structure

1. Multiple Test Cases in One File:
   • Test cases can be defined in a single file, separated by ---
   • Each test case starts with a title using # Test Case Name format
2. Directory Organization:
   • Test cases can be organized in subdirectories
   • All test cases in subdirectories are automatically discovered and executed

​

Test Case Format

# Basic Conversation Test
user: Hi there!
assistant: Hello! How can I assist you today?

---
# Weather Query Test
user: Could you check the current weather in New York for me?
assistant: It's rainy in New York today with a temperature of 59°F. There's an 80% chance of rain, high humidity (96%), and a light breeze at 2 mph. You might want to bring an umbrella!

​

Special Characters and Multiline Messages

​

Special Characters

​

Escaping Special Characters

• : is only special when it appears after a role marker
• # is only special when it appears at the start of a line
• --- is only special when it appears on its own line

# Special Characters Example
user: The price is $10:50
assistant: That's correct! The colon (:) is just part of the price.

user: Here's a markdown heading: # Important Note
assistant: The # symbol is just part of the text here.

user: The separator looks like this: ---
assistant: Yes, that's just three hyphens in the text.

​

Multiline Message Format

1. Indented Style:
   Copy

   user: This is a multiline message
         that continues on the next line
         with proper indentation
   
2. Non-indented Style:
   Copy

   user: This is another multiline message
   that continues on the next line
   without indentation
   

​

Directory Structure Example

tests/
├── basic/
│   ├── conversations.txt
│   └── simple_queries.txt
├── advanced/
│   ├── tool_usage.txt
│   └── complex_scenarios.txt
└── main.txt

​

Output Format

​

Default Output (Concise)

Passed examples/more tests/test/Silly math
Passed examples/example/First Test Case
Passed examples/more tests/test/Math
Passed examples/math/math
Passed examples/example/Second Test Case
Passed examples/example/Third Test Case
Failed examples/stocks/stocks
Passed examples/news/news
Passed examples/translation/translation
Passed examples/weather/weather

Failed Test Details
=================
examples/stocks/stocks
Status: Failed
Score: 0.398
Time: 16694.51ms

Input:
user: what's the current price of Apple stock?
assistant: I'll check the current stock price
Apple's stock (AAPL) is currently trading at USD175.25, up 2.3 percent today
user: can you write that again in simple terms?

Expected Output:
assistant: something something someting in simple terms

Actual Response:
Sure, I can rephrase that in simpler terms:

Apple's shares cost $175.25 each right now. The price went up a bit today.

Error:
Response mismatch

Test Summary
============
Total: 10
Passed: 9
Failed: 1
Duration: 50.54s

​

Example Usage

1. Basic Evaluation:
   Copy

   hebo-eval run gato-qa:v1
   
2. Custom Directory and Format:
   Copy

   hebo-eval run gato-qa:v1 -d ./my-tests -f markdown
   
3. With Configuration File:
   Copy

   hebo-eval run gato-qa:v1 -c ./hebo-evals.config.yaml
   
4. Custom Threshold and Concurrency:
   Copy

   hebo-eval run gato-qa:v1 -t 0.5 -m 10
   
5. Verbose Output:
   Copy

   hebo-eval run gato-qa:v1 -v
   

​

Best Practices

1. Always set up your API keys using environment variables for security
2. Use the provided hebo-evals.config.yaml template as a starting point
3. Start with a small test set before running large evaluations
4. Use descriptive test case titles with the # format
5. Organize test cases in subdirectories for better management
6. Keep your configuration file secure and never commit API keys to version control
7. Use the -v flag when debugging test failures
8. Leverage environment variable substitution in your configuration for better security

​

Troubleshooting

1. Verify your environment variables:
   Copy

   export HEBO_API_KEY=your_api_key_here
   
2. Or use a configuration file:
   Copy

   hebo-eval run <agent> --config path/to/hebo-evals.config.yaml