Inference Provider Testing Dashboard

Launch and monitor evaluation jobs for multiple models and providers.

📊 Total: 166 | 🔵 Running: 13 | ✅ Completed: 12 | ❌ Failed: 141

In this demo, we run 10 samples of 3 evaluations: ifeval (instruction following), gsm_plus (grade school math problems, less contaminated than gsm8k) and gpqa, diamond subset (knowledge), with lighteval, inference-providers and jobs.

The "status" column indicates whether the evaluation failed completely (usually because of the provider was down or because we were rate limited).

To run any of these locally, you can use the following

from huggingface_hub import run_job, inspect_job, whoami
job = run_job(
    image="hf.co/spaces/OpenEvals/EvalsOnTheHub",
    command=[
        "lighteval", "endpoint", "inference-providers", 
        "model_name=MODEL,provider=PROVIDER", 
        "extended|ifeval|0,lighteval|gpqa:diamond|0", 
        "--push-to-hub", "--save-details", 
        "--results-org", "YOURORG"
    ],
    namespace="huggingface",
    secrets={"HF_TOKEN": YOURTOKEN},
    token=YOURTOKEN
)