Guides

Validating your catalog in CI

Your tool catalog is configuration that changes model behaviour in production. Treat it like code: keep it in the repository, check it on every change, and deploy it deliberately.

Why this is worth ten minutes

A broken catalog does not fail loudly. Registration is rejected, so the previous catalog stays live — which is the safe behaviour and also the confusing one: your deploy succeeds, and the assistant quietly keeps offering the tools you thought you had replaced.

A catalog that registers but is subtly wrong is worse. A missing is_write is a valid catalog that executes a destructive tool inline.

Keep it in the repo

tools/
  catalog.json        the array, formatted, reviewable in a diff
  validate.sh         CI check
  register.sh         deploy step

One file, one array. A is_write flipping from true to false should be a line in a pull request that a human reads.

The check

POST /v1/tools/validate runs identical validation to registration, with no side effect.

#!/usr/bin/env bash
# tools/validate.sh
set -euo pipefail

response=$(curl -sS -X POST https://api.trytether.ge/v1/tools/validate \
  -H "X-Api-Key: $TETHER_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @tools/catalog.json)

if [ "$(echo "$response" | jq -r '.valid')" != "true" ]; then
  echo "Tool catalog is invalid:"
  echo "$response" | jq -r '.error'
  exit 1
fi

echo "$response" | jq -r '"OK — \(.toolCount) tools, \(.writeToolCount) writes"'

# .github/workflows/catalog.yml
name: tool catalog
on:
  pull_request:
    paths: ['tools/catalog.json']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./tools/validate.sh
        env:
          TETHER_KEY: ${{ secrets.TETHER_VALIDATE_KEY }}

Use a key for a staging tenant. Validation has no side effect, but a key in CI is still a key.

Assert on the write count

Structural validation cannot tell you a tool is classified correctly — only you know that. What CI can do is refuse to let the number change silently:

writes=$(echo "$response" | jq -r '.writeToolCount')
expected=$(cat tools/EXPECTED_WRITE_COUNT)

if [ "$writes" != "$expected" ]; then
  echo "Write-tool count changed: $expected -> $writes"
  echo "If this is intended, update tools/EXPECTED_WRITE_COUNT in the same commit."
  exit 1
fi

Crude, and it works. A tool silently losing its is_write flag becomes a failing build with a message naming exactly what happened, and the fix requires touching a file whose only purpose is to make you think about it.

The validate response also returns the resolved classification per tool, which is worth snapshotting if you prefer a richer diff:

{ "name": "void_ticket", "domain": "orders", "isWrite": true, "risk": "high",
  "requiresScope": false, "surfaces": { "chat": false, "agent": true } }

Local pre-commit

# .git/hooks/pre-commit
git diff --cached --name-only | grep -q '^tools/catalog.json$' && ./tools/validate.sh

Two seconds, and it catches the trailing comma before CI does.

Registering on deploy

Validation belongs in the pull request. Registration belongs in the deploy, after merge, and after the code that implements the tools is live.

#!/usr/bin/env bash
# tools/register.sh
set -euo pipefail

response=$(curl -sS -X PUT https://api.trytether.ge/v1/tools \
  -H "X-Api-Key: $TETHER_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @tools/catalog.json)

if [ "$(echo "$response" | jq -r '.registered')" != "true" ]; then
  echo "$response" | jq -r '.error'
  exit 1
fi

echo "$response" | jq -r '"Registered v\(.version): \(.toolCount) tools, \(.writeToolCount) writes"'

Verifying afterwards

curl -sS https://api.trytether.ge/v1/tools -H "X-Api-Key: $TETHER_KEY" \
  | jq '{ version, toolCount, writeToolCount, registeredAt }'

If version did not increment, your registration did not land — check the deploy log for the rejection message.

What CI cannot catch

A bad description. Structurally valid, and the model calls the tool at the wrong times. Only evaluation catches this.

A misclassified write. is_write: false on a destructive tool is a perfectly valid catalog. Only review catches this, which is why the count assertion above is worth its crudeness.

A schema that does not match your handler. Tether does not validate arguments against your implementation. A contract test on your own endpoint does.