Replay & Debugging

Debug agents with visual replays and step-by-step inspection

AI agent instructions

Read the Notte skill and follow its setup instructions when setting up Notte or building browser automation. This is the shared source for CLI setup, authentication, and the browser-to-code workflow.

Debug agent executions with MP4 replays that show exactly what the agent saw and did during its run.

Agent Replay

Get a visual replay of the agent’s execution. The session.replay() method returns a ReplayResponse with mp4_url (a presigned URL for the MP4), which you can pass around or download with replay.download():

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Navigate and extract data")

# Get MP4 replay (returns presigned URL)
replay = session.replay()
print(replay.mp4_url)  # Presigned URL for MP4 download

# Download to file
replay.download("agent_run.mp4")

What’s Included

Agent replays capture:

• Visual recording of every page the agent visited
• Actions taken by the agent (clicks, fills, navigations)
• Page transitions and loading states
• Element highlights showing what the agent interacted with
• Timestamps for each action

Viewing Replays

In Jupyter Notebook

Display the replay inline:

view_in_notebook.py

from IPython.display import Video
from notte_sdk import NotteClient

client = NotteClient()
with client.Session() as session:
    agent = client.Agent(session=session)
    agent.run(task="Complete task")

replay = session.replay()
# Display the MP4 inline in a Jupyter cell
Video(replay.mp4_url)

Save to File

Save for later viewing or sharing:

save_to_file.py

with client.Session() as session:
    agent = client.Agent(session=session)
    agent.run(task="Complete task")

replay = session.replay()
replay.download("debug_run.mp4")

# With custom path
output_dir = Path("./replays")
output_dir.mkdir(exist_ok=True)
replay.download(output_dir / f"agent_{agent.agent_id}.mp4")

In Browser

Open the replay URL in your browser:

view_in_browser.py

import webbrowser

from notte_sdk import NotteClient

client = NotteClient()
with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Complete task")

# Open the replay (presigned MP4 URL) in your browser
replay = session.replay()
if replay.mp4_url is not None:
    webbrowser.open(replay.mp4_url)

Debugging Workflow

1. Run Agent

Execute your agent task:

run_agent.py

agent = client.Agent(session=session, max_steps=15)
result = agent.run(task="Find and click the pricing button")

2. Check If Failed

Identify failures:

check_if_failed.py

agent = client.Agent(session=session)
result = agent.run(task="Complete task")

if not result.success:
    print(f"Agent failed: {result.answer}")
nb_steps = sum(1 for s in result.steps if s["type"] == "agent_step_start")
print(f"Completed {nb_steps} steps")

3. Inspect Steps

Review what the agent did:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Complete task")

    # Steps are trajectory elements: {"type": ..., "value": {...}}
    actions = [s["value"] for s in result.steps if s["type"] == "execution_result"]
    for i, step in enumerate(actions):
        print(f"Action {i + 1}: {step['action']['type']}")
        print(f"  Success: {step['success']}")
        if not step["success"]:
            print(f"  Error: {step['message']}")

4. Watch Replay

See the visual execution:

watch_replay.py

with client.Session() as session:
    agent = client.Agent(session=session)
    agent.run(task="Complete task")

replay = session.replay()
replay.download("debug_replay.mp4")

# Watch where it failed
# Identify if element selectors were wrong
# Check if page loaded correctly

5. Fix and Retry

Adjust based on findings:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    # Maybe increase max_steps
    agent = client.Agent(session=session, max_steps=25)

    # Or adjust the task
    result = agent.run(task="Click the 'View Pricing' button in the main navigation")

Common Issues

Agent Can’t Find Element

Symptoms:

• Agent fails with “Element not found”
• Replay shows element exists but wasn’t clicked

Debug:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Complete task")

replay = session.replay()
replay.download("missing_element.mp4")

# Watch replay and check:
# - Did page finish loading?
# - Is element visible/in viewport?
# - Is element covered by another element?

Fix:

fix_missing_element.py

agent = client.Agent(session=session)
# Be more specific in task
task = "Wait for the page to load, then click the blue 'Submit' button at the bottom"

# Or use session actions first
session.execute(type="goto", url="https://example.com")
time.sleep(2)  # Wait for load
result = agent.run(task=task)

Agent Takes Wrong Action

Symptoms:

• Agent clicks wrong button
• Agent fills wrong form field

Debug:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Fill the email field")

    # Check what the agent saw and decided at each step
    for step in result.steps:
        if step["type"] != "agent_completion":
            continue
        completion = step["value"]
        print(f"Page summary: {completion['state']['page_summary']}")
        print(f"Next goal: {completion['state']['next_goal']}")
        print(f"Action: {completion['action']}")

Fix:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    # Be more specific
    task = "Fill the email input field in the signup form with [e-mail]"

    # Or disable vision if causing confusion
    agent = client.Agent(session=session, use_vision=False)
    result = agent.run(task=task)

Agent Exceeds Max Steps

Symptoms:

• Agent stops before completing task
• result.success is False and the number of agent_step_start entries in result.steps equals max_steps

Debug:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Complete task")

    nb_steps = sum(1 for s in result.steps if s["type"] == "agent_step_start")
    print(f"Steps taken: {nb_steps}")
    print(f"Max steps: {agent.request.max_steps}")

replay = session.replay()
replay.download("debug_replay.mp4")  # See where it got stuck

Fix:

from notte_sdk import NotteClient

client = NotteClient()

with client.Session() as session:
    # Increase max_steps
    agent = client.Agent(session=session, max_steps=30)

    # Or break task into smaller steps
    result1 = agent.run(task="Navigate to products page")
    result2 = agent.run(task="Search for laptops")

Advanced Debugging

Step-by-Step Analysis

Inspect each step in detail:

step_by_step_analysis.py

result = agent.run(task="Complex task")

for step in result.steps:
    value = step["value"]
    if step["type"] == "agent_completion":
        state = value["state"]
        print("\n=== Agent decision ===")
        print(f"Previous goal: {state['previous_goal_status']} - {state['previous_goal_eval']}")
        print(f"Next goal: {state['next_goal']}")
    elif step["type"] == "execution_result":
        action = value["action"]
        print(f"Action: {action['type']}")
        if action["type"] == "goto":
            print(f"URL: {action['url']}")
        print(f"Success: {value['success']}")
        print(f"Message: {value['message']}")

Comparing Runs

Compare successful vs failed runs:

from notte_sdk import NotteClient

client = NotteClient()

# Successful run
with client.Session() as session_1:
    agent_1 = client.Agent(session=session_1)
    result_success = agent_1.run(task="Working task")
replay_success = session_1.replay()
replay_success.download("success.mp4")

# Failed run
with client.Session() as session_2:
    agent_2 = client.Agent(session=session_2)
    result_fail = agent_2.run(task="Failing task")
replay_fail = session_2.replay()
replay_fail.download("failure.mp4")

# Compare side-by-side
# Identify where behavior diverges

Agent Status During Execution

Monitor agent in real-time:

monitor_status.py

with client.Session() as session:
    agent = client.Agent(session=session)
    agent.start(task="Long running task")

    while True:
        status = agent.status()

        if status.status == "closed":
            break

        nb_steps = sum(1 for s in status.steps if s["type"] == "agent_step_stop")
        print(f"Steps completed: {nb_steps}")

        time.sleep(5)

# Get replay after completion
# replay = session.replay()
# replay.download("replay.mp4")

Best Practices

1. Always Review Failed Agents

Don’t ignore failures - watch the replay:

review_failed_agents.py

with client.Session() as session:
    agent = client.Agent(session=session)
    result = agent.run(task="Critical task")

if not result.success:
    replay = session.replay()
    replay.download(f"failure_{agent.agent_id}.mp4")
    print(f"Saved replay for debugging: failure_{agent.agent_id}.mp4")

2. Save Replays for Production Issues

Keep replays of production failures:

save_production_replays.py

    result = agent.run(task="Production task")
    if not result.success:
        replay = session.replay()
        replay.download(f"prod_failure_{agent.agent_id}.mp4")
        logger.error(f"Agent failed, replay saved: prod_failure_{agent.agent_id}.mp4")
except Exception as e:
    logger.error(f"Agent exception: {e}")

3. Use Replays for QA

Verify agent behavior before deploying:

replays_for_qa.py

with client.Session() as session:
    agent = client.Agent(session=session)
    # Test run
    result = agent.run(task="Test checkout flow")

# Review replay before production
replay = session.replay()
replay.download("qa_replay.mp4")

# Verify:
# - All steps completed correctly
# - No unexpected behavior
# - Performance is acceptable

4. Share Replays with Team

Save and share for collaboration:

share_with_team.py

with client.Session() as session:
    agent = client.Agent(session=session)
    agent.run(task="Feature test")

replay = session.replay()
replay.download(f"feature_test_{agent.agent_id}.mp4")

# Share the file with your team
# Or share the presigned URL (valid until replay.expires_at)
print(f"Replay: {replay.mp4_url} (expires {replay.expires_at})")

Limitations

Replays are currently:

• ✅ Available for all completed agents
• ✅ Full video recording of execution
• ✅ Include element highlights
• ⚠️ Generated after completion (not real-time)
• ⚠️ May be large files for long runs

Next Steps

Lifecycle

Understand agent execution

Configuration

Optimize agent settings

Live View

Watch sessions in real-time