Robodex

The interface could display more confidence than the backend earned

The workspace generates candidate robot morphologies and renders them in 3D. During review, I found that several visible signals were not qualification-grade: Candidate A was selected by default, multiple cards could project the same selected graph, and the workspace’s “simulation check” read stored flags instead of executing physics.

That audit changed the technical question. The evaluator now asks whether a final robot artifact passes checks that are outside the producing agent’s transcript and cannot be rewritten after the run.

What is frozen, what executes, and what gets compared

The file manifest rejects aliases before hashing bytes

A digest is useful only if the tree cannot smuggle in mutable aliases. The compact control implementation rejects symlinks, non-regular files, and hard links before it records each relative path, byte count, and hash.

def _artifact_manifest(root: Path) -> list[dict[str, Any]]:
    resolved = root.resolve()
    rows = []
    for path in sorted(resolved.rglob("*")):
        if path.is_symlink():
            raise ValueError(f"artifact tree contains symlink: {path}")
        metadata = path.lstat()
        if stat.S_ISDIR(metadata.st_mode):
            continue
        if not stat.S_ISREG(metadata.st_mode):
            raise ValueError(f"artifact tree contains non-regular file: {path}")
        if metadata.st_nlink > 1:
            raise ValueError(f"artifact tree contains hard link: {path}")
        data = path.read_bytes()
        rows.append({
            "path": path.relative_to(resolved).as_posix(),
            "sha256": _sha256_bytes(data),
            "size_bytes": len(data),
        })
    return rows

Replay hashes the grading basis, not log formatting

The replay digest includes the task, grader, artifact digest, outcome state, failed grade IDs, and grade payload. It deliberately excludes fields such as wall-clock timestamps that would make identical evidence look different.

def _replay_basis(record):
    keys = (
        "task_id", "task_version", "task_digest",
        "grader_digest", "artifact_digest", "state",
        "failed_grade_ids", "grades",
    )
    return {key: record.get(key) for key in keys}

digests = [_sha256_json(_replay_basis(result)) for result in results]
identical = len(set(digests)) == 1
matches_record = all(digest == original_digest for digest in digests)

Two result sets with different scopes

Sources: the local POC report at commit 3efa712 and the saved bounded LangSmith export. The public product repository does not yet contain the eval branch. Fixture results validate the evaluator; they are not live agent performance.

What the current evidence does not establish

• The live two-profile Codex experiment is unrun: the inner sandbox could not start inside the outer macOS sandbox, and global skill descriptions contaminated the intended empty profile.
• The screenshot proves that the workspace renders and exposes its degraded mode. It does not prove that the visible candidates are physically distinct or validated.
• Passing reference and seeded-failure fixtures test grader behavior; they do not measure an agent’s ability to repair unseen robots.
• The 291-SLOC control matching the full evaluator weakens any claim that the harness itself is defensible product differentiation.