Step 1: Create Task#

A task defines one or more instructions, a sandbox environment, and a verifier. Tasks are used to evaluate agents and models and are implemented as directories in the Harbor task format.

Think of Harbor task as standardized exam for AI agents that includes:

• the question: The prompt or problem statement given to the AI
• the workbench: an isolated container with necessary software installed as a clean workspace provided for the task

Run the following command to create a new task directory with the required files:

1
harbor task init ssh-key-pair

We will evaluate/benchmark an agent or model using task like this.

Step 2: Writing Task Instructions#

Open the instruction.md file in the task directory and add the task description:

1
# SSH Key Pair Generation
2

3
Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`.
4

5
Don't make them password protected.

Think of instruction.md as the “exam question”. In this example, our agent will perform a task defined by this instruction.md file and generate some verifiable results as the agent’s “answer sheet to the exam”.

Step 3: Configuring Task Metadata#

Open the task.toml file to configure task metadata, resource limits, and timeouts.

NOTE

Most fields in task.toml are optional, allowing Harbor to defer configuration to our environment files unless specific overrides or metadata are needed.

1
version = "1.0"
2

3
[metadata]
4
author_name = "Your Name"
5
author_email = "your.email@example.com"
6
difficulty_explanation = "Simple SSH key generation command"
7
category = "system-administration"
8
tags = ["ssh", "cryptography", "linux"]
9

10
[verifier]
11
timeout_sec = 120.0
12

13
[agent]
14
timeout_sec = 120.0
15

16
[environment]
17
build_timeout_sec = 600.0

Step 4: Creating Task Container In Which Agent Works#

Now that we have exam sheet, let’s create a test room for our agent. A container Dockerfile defines such an environment an agent will interact with through the terminal. Open the Dockerfile in the environment/ directory that was generated and add any dependencies our task needs:

1
FROM ubuntu:24.04
2

3
# Create working directory
4
WORKDIR /app
5

6
# Install openssh-client for the task
7
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*

Harbor spins up the container using this Dockerfile and the AI agent is injected inside it. The agent will interact with this container environment via a simulated terminal - running shell commands, installing tools, and editing files right inside the container

TIP

The AI agent under test runs inside the container.

Step 5: Creating Solution Script#

Proof of Solvability (The “Oracle” Agent)

Before benchmarking real AI models, we need to verify that our task isn’t broken (e.g., checking that required dependencies like openssh-client aren’t missing from the container).

To do that, Harbor runs an Oracle Agent, which simply executes some script called solution script inside the test container followed by the test script to verify the outcome.

If such script passes, we can prove that our environment (“test room”) is good for the agent to solve the task (“finish the exam”)

We implement this reference solution in ssh-key-pair/solution/solve.sh below:

1
#!/bin/bash
2

3
ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""

We also want to make sure the script is executable:

1
chmod +x ssh-key-pair/solution/solve.sh

Now we can run the following command to verify our task is solvable by the solution script:

1
cd ../ # navigate to the directory that contains the "ssh-key-pair" task folder
2
harbor run -p ssh-key-pair -a oracle

If successful, we should see output indicating the task was completed and the reward was 1.

Step 6: Creating Test Script#

Now that the tested agent has “finished the exam”, we need to grade the agent’s answer, i.e. whether the agent has successfully completed the task. We do this via test scripts, which consist of 2 parts:

1. A language-agnostic test harness that “grades agent’s answer”
2. An executor that runs the test harness and must be written in bash named test.sh

We first write the actual tests, for example in Python for simplicity:

1
import os
2
from pathlib import Path
3

4

5
def test_key_files_exist() -> None:
6
    """Test that both private and public key files exist."""
7
    private_key = Path.home() / ".ssh" / "id_rsa"
8
    public_key = Path.home() / ".ssh" / "id_rsa.pub"
9

10
    assert private_key.exists(), "Private key file does not exist"
11
    assert public_key.exists(), "Public key file does not exist"
12

13

14
def test_key_file_permissions() -> None:
15
    """Test that the key files have correct permissions."""
16
    private_key = Path.home() / ".ssh" / "id_rsa"
17
    public_key = Path.home() / ".ssh" / "id_rsa.pub"
18

19
    private_perms = oct(os.stat(private_key).st_mode)[-3:]
20
    public_perms = oct(os.stat(public_key).st_mode)[-3:]
21

22
    assert private_perms == "600", (
23
        f"Private key has incorrect permissions: {private_perms}"
24
    )
25
    assert public_perms == "644", (
26
        f"Public key has incorrect permissions: {public_perms}"
27
    )
28

29

30
def test_key_format() -> None:
31
    """Test that the public key has the correct RSA format."""
32
    public_key = Path.home() / ".ssh" / "id_rsa.pub"
33

34
    with open(public_key, 'r') as f:
35
        content = f.read()
36

37
    assert content.startswith("ssh-rsa "), "Public key does not start with 'ssh-rsa'"
38
    assert len(content.split()) >= 2, "Public key format is invalid"

Next, create the execution wrapper that must be written in Bash and named test.sh:

IMPORTANT

The script must write a numerical reward (1 for pass, 0 for fail) to /logs/verifier/reward.txt. This is how Harbor communicates whether the agent succeeded.

1
#!/bin/bash
2

3
apt-get update
4
apt-get install -y curl
5

6
curl -LsSf https://astral.sh/uv/0.9.5/install.sh | sh
7

8
source $HOME/.local/bin/env
9

10
# Run pytest tests
11
uvx \
12
  --python 3.12 \
13
  --with pytest==8.4.1 \
14
  pytest /tests/test_outputs.py
15

16
# Check exit code and write reward
17
if [ $? -eq 0 ]; then
18
  echo 1 > /logs/verifier/reward.txt
19
else
20
  echo 0 > /logs/verifier/reward.txt
21
fi