Skip to content

Commit 1dd7b23

Browse files
cexllcexll
committed
feat harness skill
1 parent 664d827 commit 1dd7b23

1 file changed

Lines changed: 329 additions & 0 deletions

File tree

‎skills/harness/SKILL.md‎

Lines changed: 329 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,329 @@
1+
---
2+
name: harness
3+
description: "This skill should be used for multi-session autonomous agent work requiring progress checkpointing, failure recovery, and task dependency management. Triggers on '/harness' command, or when a task involves many subtasks needing progress persistence, sleep/resume cycles across context windows, recovery from mid-task failures with partial state, or distributed work across multiple agent sessions. Synthesized from Anthropic and OpenAI engineering practices for long-running agents."
4+
---
5+
6+
# Harness — Long-Running Agent Framework
7+
8+
Executable protocol enabling any agent task to run continuously across multiple sessions with automatic progress recovery, task dependency resolution, failure rollback, and standardized error handling.
9+
10+
## Design Principles
11+
12+
1. **Design for the agent, not the human** — Test output, docs, and task structure are the agent's primary interface
13+
2. **Progress files ARE the context** — When context window resets, progress files + git history = full recovery
14+
3. **Premature completion is the #1 failure mode** — Structured task lists with explicit completion criteria prevent declaring victory early
15+
4. **Standardize everything grep-able** — ERROR on same line, structured timestamps, consistent prefixes
16+
5. **Fast feedback loops** — Pre-compute stats, run smoke tests before full validation
17+
6. **Idempotent everything** — Init scripts, task execution, environment setup must all be safe to re-run
18+
7. **Fail safe, not fail silent** — Every failure must have an explicit recovery strategy
19+
20+
## Commands
21+
22+
```
23+
/harness init <project-path> # Initialize harness files in project
24+
/harness run # Start/resume the infinite loop
25+
/harness status # Show current progress and stats
26+
/harness add "task description" # Add a task to the list
27+
```
28+
29+
## Progress Persistence (Dual-File System)
30+
31+
Maintain two files in the project working directory:
32+
33+
### harness-progress.txt (Append-Only Log)
34+
35+
Free-text log of all agent actions across sessions. Never truncate.
36+
37+
```
38+
[2025-07-01T10:00:00Z] [SESSION-1] INIT Harness initialized for project /path/to/project
39+
[2025-07-01T10:00:05Z] [SESSION-1] INIT Environment health check: PASS
40+
[2025-07-01T10:00:10Z] [SESSION-1] LOCK acquired (pid=12345)
41+
[2025-07-01T10:00:11Z] [SESSION-1] Starting [task-001] Implement user authentication (base=def5678)
42+
[2025-07-01T10:05:00Z] [SESSION-1] CHECKPOINT [task-001] step=2/4 "auth routes created, tests pending"
43+
[2025-07-01T10:15:30Z] [SESSION-1] Completed [task-001] (commit abc1234)
44+
[2025-07-01T10:15:31Z] [SESSION-1] Starting [task-002] Add rate limiting (base=abc1234)
45+
[2025-07-01T10:20:00Z] [SESSION-1] ERROR [task-002] [TASK_EXEC] Redis connection refused
46+
[2025-07-01T10:20:01Z] [SESSION-1] ROLLBACK [task-002] git reset --hard abc1234
47+
[2025-07-01T10:20:02Z] [SESSION-1] STATS tasks_total=5 completed=1 failed=1 pending=3 blocked=0 attempts_total=2 checkpoints=1
48+
```
49+
50+
### harness-tasks.json (Structured State)
51+
52+
```json
53+
{
54+
"version": 2,
55+
"created": "2025-07-01T10:00:00Z",
56+
"session_config": {
57+
"max_tasks_per_session": 20,
58+
"max_sessions": 50
59+
},
60+
"tasks": [
61+
{
62+
"id": "task-001",
63+
"title": "Implement user authentication",
64+
"status": "completed",
65+
"priority": "P0",
66+
"depends_on": [],
67+
"attempts": 1,
68+
"max_attempts": 3,
69+
"started_at_commit": "def5678",
70+
"validation": {
71+
"command": "npm test -- --testPathPattern=auth",
72+
"timeout_seconds": 300
73+
},
74+
"on_failure": {
75+
"cleanup": null
76+
},
77+
"error_log": [],
78+
"checkpoints": [],
79+
"completed_at": "2025-07-01T10:15:30Z"
80+
},
81+
{
82+
"id": "task-002",
83+
"title": "Add rate limiting",
84+
"status": "failed",
85+
"priority": "P1",
86+
"depends_on": [],
87+
"attempts": 1,
88+
"max_attempts": 3,
89+
"started_at_commit": "abc1234",
90+
"validation": {
91+
"command": "npm test -- --testPathPattern=rate-limit",
92+
"timeout_seconds": 120
93+
},
94+
"on_failure": {
95+
"cleanup": "docker compose down redis"
96+
},
97+
"error_log": ["[TASK_EXEC] Redis connection refused"],
98+
"checkpoints": [],
99+
"completed_at": null
100+
},
101+
{
102+
"id": "task-003",
103+
"title": "Add OAuth providers",
104+
"status": "pending",
105+
"priority": "P1",
106+
"depends_on": ["task-001"],
107+
"attempts": 0,
108+
"max_attempts": 3,
109+
"started_at_commit": null,
110+
"validation": {
111+
"command": "npm test -- --testPathPattern=oauth",
112+
"timeout_seconds": 180
113+
},
114+
"on_failure": {
115+
"cleanup": null
116+
},
117+
"error_log": [],
118+
"checkpoints": [],
119+
"completed_at": null
120+
}
121+
],
122+
"session_count": 1,
123+
"last_session": "2025-07-01T10:20:02Z"
124+
}
125+
```
126+
127+
Task statuses: `pending` → `in_progress` (transient, set only during active execution) → `completed` or `failed`. A task found as `in_progress` at session start means the previous session was interrupted — handle via Context Window Recovery Protocol.
128+
129+
**Session boundary**: A session starts when the agent begins executing the Session Start protocol and ends when a Stopping Condition is met or the context window resets. Each session gets a unique `SESSION-N` identifier (N = `session_count` after increment).
130+
131+
## Concurrency Control
132+
133+
Before modifying `harness-tasks.json`, acquire an exclusive lock using portable `mkdir` (atomic on all POSIX systems, works on both macOS and Linux):
134+
135+
```bash
136+
# Acquire lock (fail fast if another agent is running)
137+
LOCKDIR="/tmp/harness-$(printf '%s' "$(pwd)" | shasum -a 256 2>/dev/null || sha256sum | cut -c1-8).lock"
138+
if ! mkdir "$LOCKDIR" 2>/dev/null; then
139+
# Check if lock holder is still alive
140+
LOCK_PID=$(cat "$LOCKDIR/pid" 2>/dev/null)
141+
if [ -n "$LOCK_PID" ] && kill -0 "$LOCK_PID" 2>/dev/null; then
142+
echo "ERROR: Another harness session is active (pid=$LOCK_PID)"; exit 1
143+
fi
144+
# Stale lock — atomically reclaim via mv to avoid TOCTOU race
145+
STALE="$LOCKDIR.stale.$$"
146+
if mv "$LOCKDIR" "$STALE" 2>/dev/null; then
147+
rm -rf "$STALE"
148+
mkdir "$LOCKDIR" || { echo "ERROR: Lock contention"; exit 1; }
149+
echo "WARN: Removed stale lock${LOCK_PID:+ from pid=$LOCK_PID}"
150+
else
151+
echo "ERROR: Another agent reclaimed the lock"; exit 1
152+
fi
153+
fi
154+
echo "$$" > "$LOCKDIR/pid"
155+
trap 'rm -rf "$LOCKDIR"' EXIT
156+
```
157+
158+
Log lock acquisition: `[timestamp] [SESSION-N] LOCK acquired (pid=<PID>)`
159+
Log lock release: `[timestamp] [SESSION-N] LOCK released`
160+
161+
The lock is held for the entire session. The `trap EXIT` handler releases it automatically on normal exit, errors, or signals. Never release the lock between tasks within a session.
162+
163+
## Infinite Loop Protocol
164+
165+
### Session Start (Execute Every Time)
166+
167+
1. **Read state**: Read last 200 lines of `harness-progress.txt` + full `harness-tasks.json`. If JSON is unparseable, see JSON corruption recovery in Error Handling.
168+
2. **Read git**: Run `git log --oneline -20` and `git diff --stat` to detect uncommitted work
169+
3. **Acquire lock**: Fail if another session is active
170+
4. **Recover interrupted tasks** (see Context Window Recovery below)
171+
5. **Health check**: Run `harness-init.sh` if it exists
172+
6. **Track session**: Increment `session_count` in JSON. Check `session_count` against `max_sessions` — if reached, log STATS and STOP. Initialize per-session task counter to 0.
173+
7. **Pick next task** using Task Selection Algorithm below
174+
175+
### Task Selection Algorithm
176+
177+
Before selecting, run dependency validation:
178+
179+
1. **Cycle detection**: For each non-completed task, walk `depends_on` transitively. If any task appears in its own chain, mark it `failed` with `[DEPENDENCY] Circular dependency detected: task-A -> task-B -> task-A`. Self-references (`depends_on` includes own id) are also cycles.
180+
2. **Blocked propagation**: If a task's `depends_on` includes a task that is `failed` and will never be retried (either `attempts >= max_attempts` OR its `error_log` contains a `[DEPENDENCY]` entry), mark the blocked task as `failed` with `[DEPENDENCY] Blocked by failed task-XXX`. Repeat until no more tasks can be propagated.
181+
182+
Then pick the next task in this priority order:
183+
184+
1. Tasks with `status: "pending"` where ALL `depends_on` tasks are `completed` — sorted by `priority` (P0 > P1 > P2), then by `id` (lowest first)
185+
2. Tasks with `status: "failed"` where `attempts < max_attempts` and ALL `depends_on` are `completed` — sorted by priority, then oldest failure first
186+
3. If no eligible tasks remain → log final STATS → STOP
187+
188+
### Task Execution Cycle
189+
190+
For each task, execute this exact sequence:
191+
192+
1. **Claim**: Record `started_at_commit` = current HEAD hash. Set status to `in_progress`, log `Starting [<task-id>] <title> (base=<hash>)`
193+
2. **Execute with checkpoints**: Perform the work. After each significant step, log:
194+
```
195+
[timestamp] [SESSION-N] CHECKPOINT [task-id] step=M/N "description of what was done"
196+
```
197+
Also append to the task's `checkpoints` array: `{ "step": M, "total": N, "description": "...", "timestamp": "ISO" }`
198+
3. **Validate**: Run the task's `validation.command` wrapped with `timeout`: `timeout <timeout_seconds> <command>`. If no validation command, skip. Before running, verify the command exists (e.g., `command -v <binary>`) — if missing, treat as `ENV_SETUP` error.
199+
- Command exits 0 → PASS
200+
- Command exits non-zero → FAIL
201+
- Command exceeds timeout → TIMEOUT
202+
4. **Record outcome**:
203+
- **Success**: status=`completed`, set `completed_at`, log `Completed [<task-id>] (commit <hash>)`, git commit
204+
- **Failure**: increment `attempts`, append error to `error_log`. Verify `started_at_commit` exists via `git cat-file -t <hash>` — if missing, mark failed at max_attempts. Otherwise execute `git reset --hard <started_at_commit>` and `git clean -fd` to rollback ALL commits and remove untracked files. Execute `on_failure.cleanup` if defined. Log `ERROR [<task-id>] [<category>] <message>`. Set status=`failed` (Task Selection Algorithm pass 2 handles retries when attempts < max_attempts)
205+
5. **Track**: Increment per-session task counter. If `max_tasks_per_session` reached, log STATS and STOP.
206+
6. **Continue**: Immediately pick next task (zero idle time)
207+
208+
### Stopping Conditions
209+
210+
- All tasks `completed`
211+
- All remaining tasks `failed` at max_attempts or blocked by failed dependencies
212+
- `session_config.max_tasks_per_session` reached for this session
213+
- `session_config.max_sessions` reached across all sessions
214+
- User interrupts
215+
216+
## Context Window Recovery Protocol
217+
218+
When a new session starts and finds a task with `status: "in_progress"`:
219+
220+
1. **Check git state**:
221+
```bash
222+
git diff --stat # Uncommitted changes?
223+
git log --oneline -5 # Recent commits since task started?
224+
git stash list # Any stashed work?
225+
```
226+
2. **Check checkpoints**: Read the task's `checkpoints` array to determine last completed step
227+
3. **Decision matrix** (verify recent commits belong to this task by checking commit messages for the task-id):
228+
229+
| Uncommitted? | Recent task commits? | Checkpoints? | Action |
230+
|---|---|---|---|
231+
| No | No | None | Mark `failed` with `[SESSION_TIMEOUT] No progress detected`, increment attempts |
232+
| No | No | Some | Verify file state matches checkpoint claims. If files reflect checkpoint progress, resume from last step. If not, mark `failed` — work was lost |
233+
| No | Yes | Any | Run `validation.command`. If passes → mark `completed`. If fails → `git reset --hard <started_at_commit>`, mark `failed` |
234+
| Yes | No | Any | Run validation WITH uncommitted changes present. If passes → commit, mark `completed`. If fails → `git reset --hard <started_at_commit>` + `git clean -fd`, mark `failed` |
235+
| Yes | Yes | Any | Commit uncommitted changes, run `validation.command`. If passes → mark `completed`. If fails → `git reset --hard <started_at_commit>` + `git clean -fd`, mark `failed` |
236+
237+
4. **Log recovery**: `[timestamp] [SESSION-N] RECOVERY [task-id] action="<action taken>" reason="<reason>"`
238+
239+
## Error Handling & Recovery Strategies
240+
241+
Each error category has a default recovery strategy:
242+
243+
| Category | Default Recovery | Agent Action |
244+
|----------|-----------------|--------------|
245+
| `ENV_SETUP` | Re-run init, then STOP if still failing | Run `harness-init.sh` again immediately. If fails twice, log and stop — environment is broken |
246+
| `TASK_EXEC` | Rollback via `git reset --hard <started_at_commit>`, retry | Verify `started_at_commit` exists (`git cat-file -t <hash>`). If missing, mark failed at max_attempts. Otherwise reset, run `on_failure.cleanup` if defined, retry if attempts < max_attempts |
247+
| `TEST_FAIL` | Rollback via `git reset --hard <started_at_commit>`, retry | Reset to `started_at_commit`, analyze test output to identify fix, retry with targeted changes |
248+
| `TIMEOUT` | Kill process, execute cleanup, retry | Wrap validation with `timeout <seconds> <command>`. On timeout, run `on_failure.cleanup`, retry (consider splitting task if repeated) |
249+
| `DEPENDENCY` | Skip task, mark blocked | Log which dependency failed, mark task as `failed` with dependency reason |
250+
| `SESSION_TIMEOUT` | Use Context Window Recovery Protocol | New session assesses partial progress via Recovery Protocol — may result in completion or failure depending on validation |
251+
252+
**JSON corruption**: If `harness-tasks.json` cannot be parsed, check for `harness-tasks.json.bak` (written before each modification). If backup exists and is valid, restore from it. If no valid backup, log `ERROR [ENV_SETUP] harness-tasks.json corrupted and unrecoverable` and STOP — task metadata (validation commands, dependencies, cleanup) cannot be reconstructed from logs alone.
253+
254+
**Backup protocol**: Before every write to `harness-tasks.json`, copy the current file to `harness-tasks.json.bak`.
255+
256+
## Environment Initialization
257+
258+
If `harness-init.sh` exists in the project root, run it at every session start. The script must be idempotent.
259+
260+
Example `harness-init.sh`:
261+
```bash
262+
#!/bin/bash
263+
set -e
264+
npm install 2>/dev/null || pip install -r requirements.txt 2>/dev/null || true
265+
curl -sf http://localhost:5432 >/dev/null 2>&1 || echo "WARN: DB not reachable"
266+
npm test -- --bail --silent 2>/dev/null || echo "WARN: Smoke test failed"
267+
echo "Environment health check complete"
268+
```
269+
270+
## Standardized Log Format
271+
272+
All log entries use grep-friendly format on a single line:
273+
274+
```
275+
[ISO-timestamp] [SESSION-N] <TYPE> [task-id]? [category]? message
276+
```
277+
278+
`[task-id]` and `[category]` are included when applicable (task-scoped entries). Session-level entries (`INIT`, `LOCK`, `STATS`) omit them.
279+
280+
Types: `INIT`, `Starting`, `Completed`, `ERROR`, `CHECKPOINT`, `ROLLBACK`, `RECOVERY`, `STATS`, `LOCK`, `WARN`
281+
282+
Error categories: `ENV_SETUP`, `TASK_EXEC`, `TEST_FAIL`, `TIMEOUT`, `DEPENDENCY`, `SESSION_TIMEOUT`
283+
284+
Filtering:
285+
```bash
286+
grep "ERROR" harness-progress.txt # All errors
287+
grep "ERROR" harness-progress.txt | grep "TASK_EXEC" # Execution errors only
288+
grep "SESSION-3" harness-progress.txt # All session 3 activity
289+
grep "STATS" harness-progress.txt # All session summaries
290+
grep "CHECKPOINT" harness-progress.txt # All checkpoints
291+
grep "RECOVERY" harness-progress.txt # All recovery actions
292+
```
293+
294+
## Session Statistics
295+
296+
At session end, update `harness-tasks.json`: increment `session_count`, set `last_session` to current timestamp. Then append:
297+
298+
```
299+
[timestamp] [SESSION-N] STATS tasks_total=10 completed=7 failed=1 pending=2 blocked=0 attempts_total=12 checkpoints=23
300+
```
301+
302+
`blocked` is computed at stats time: count of pending tasks whose `depends_on` includes a permanently failed task. It is not a stored status value.
303+
304+
## Init Command (`/harness init`)
305+
306+
1. Create `harness-progress.txt` with initialization entry
307+
2. Create `harness-tasks.json` with empty task list and default `session_config`
308+
3. Optionally create `harness-init.sh` template (chmod +x)
309+
4. Ask user: add harness files to `.gitignore`?
310+
311+
## Status Command (`/harness status`)
312+
313+
Read `harness-tasks.json` and `harness-progress.txt`, then display:
314+
315+
1. Task summary: count by status (completed, failed, pending, blocked). `blocked` = pending tasks whose `depends_on` includes a permanently failed task (computed, not a stored status).
316+
2. Per-task one-liner: `[status] task-id: title (attempts/max_attempts)`
317+
3. Last 5 lines from `harness-progress.txt`
318+
4. Session count and last session timestamp
319+
320+
Does NOT acquire the lock (read-only operation).
321+
322+
## Add Command (`/harness add`)
323+
324+
Append a new task to `harness-tasks.json` with auto-incremented id (`task-NNN`), status `pending`, default `max_attempts: 3`, empty `depends_on`, and no validation command. Prompt user for optional fields: `priority`, `depends_on`, `validation.command`, `timeout_seconds`. Requires lock acquisition (modifies JSON).
325+
326+
## Tool Dependencies
327+
328+
Requires: Bash, file read/write, git. All harness operations must be executed from the project root directory.
329+
Does NOT require: specific MCP servers, programming languages, or test frameworks.

0 commit comments

Comments
 (0)