What actually survives: closing the persistence gap

Read commit 054ea75 on GitHub · chuyeowego/not-your-slave

Background

Deep background: where an agent's shell actually runs

Give a model a bash tool and the immediate question is whose machine it runs on. eve's answer is a sandbox: an isolated environment rooted at /workspace, with a backend chosen by availability - Vercel Sandbox when deployed on Vercel, otherwise Docker if a daemon is reachable, otherwise a lightweight local VM, otherwise a pure-JS Bash interpreter.

On this machine it resolves to Docker, and the isolation is worth stating in the concrete terms it was checked in, because the alternative is trusting a diagram:

$ docker inspect <sandbox> --format 'Mounts: {{json .Mounts}} …'
Mounts: []
Network: bridge
Privileged: false
CapAdd: null

# and from inside the container:
$ hostname; whoami; ls /Users
e91edb4887c0
vercel-sandbox
ls: cannot access '/Users': No such file or directory

No mounts, no privileges, no host filesystem. The agent's shell cannot read your home directory, your repo, or your keys. It does have open network egress, which is a separate conversation and one line of policy to change.

Narrow background: three places state can live

That third row is the gap. Commit 4 established that a markdown schedule starts a new session on every fire, and now those two facts meet.

Intuition

Two candidate fixes, and the choice between them is about whose problem it is.

• Give the agent host-folder tools - a keep_file / read_kept pair writing to .data/workspace/. Ten minutes of work. It also gives the agent two filesystems and a rule to remember, and it will forget, because remembering to copy your work into the durable drive is exactly the kind of bookkeeping commit 3 already established a model will skip.
• Sync at the session boundary - restore on start, snapshot on park. More code, in one place, and the agent's mental model stays "I use bash on /workspace". Nothing to remember, so nothing to forget.

Same reasoning as commit 3: put the mechanism in the runtime rather than in the prompt. The framework's Docker backend exposes image, env, pull policy and network policy - but no volumes - so crossing the boundary means copying bytes, and the cheapest copy of a directory tree is a tarball.

The measurement

Both the gap and the fix were measured rather than reasoned about, and the evidence is worth reproducing because it is what makes the claims checkable.

The gap, in dev

The gap, under real cron

A temporary schedule at * * * * *, whose prompt appended a line to a file and reported the count:

07:08:03  FIRES=1  container 5804840ac060  session …TSK950
07:09:11  FIRES=2  container 0d89c9f5b6e1  session …MCSF0M

Two fires, two sessions, two containers. Before the hook the second fire found no file at all; the run above is after, and the hook's own log lines are the other half of the evidence:

[workspace-sync] wrun_…TSK950  nothing kept yet
[workspace-sync] wrun_…TSK950  kept 163 bytes
[workspace-sync] wrun_…MCSF0M  restored 163 bytes
[workspace-sync] wrun_…MCSF0M  kept 170 bytes

What else was found while looking

Counting what is stored turned up something unrelated and worse: .eve/.workflow-data/streams had grown to 2.6 GB, of which 2.57 GB belonged to one session - 31,435 chunks averaging 86 KB. The cause is the cumulative-delta design from commit 5: every message.appended stores the whole message so far, so a long reply is persisted repeatedly at growing length. It is framework behaviour, not a bug in this code, and it is almost certainly what OOM-killed the dev server earlier in the session at a 4 GB heap. The practical consequence is that wiping .eve/ becomes routine - which is precisely why it mattered that the mindlog and the workspace archive lived somewhere else.

The code

const ARCHIVE = process.env.WORKSPACE_ARCHIVE ?? ".data/workspace.tgz";
const SANDBOX_ARCHIVE = "/tmp/workspace-sync.tgz";
const MINDLOG_COPY = "/workspace/mindlog.jsonl";

async function restore(sandbox: SandboxSession): Promise<string> {
  let archive: Uint8Array;
  try { archive = await readFile(ARCHIVE); } catch { return "nothing kept yet"; }

  await sandbox.writeBinaryFile({ path: SANDBOX_ARCHIVE, content: archive });
  const result = await sandbox.run({
    command: `mkdir -p /workspace && tar xzf ${SANDBOX_ARCHIVE} -C /workspace && rm -f ${SANDBOX_ARCHIVE}`,
  });
  return result.exitCode === 0 ? `restored ${archive.byteLength} bytes` : `restore failed: ${result.stderr}`;
}

Bytes in through the sandbox handle, then tar inside the container does the unpacking. A missing archive is not an error - it is a new agent - and returns a string that the caller logs, which is how the evidence above exists at all.

async function snapshot(sandbox: SandboxSession): Promise<string> {
  // Exclude the mindlog copy: the real one lives on the host and is the source
  // of truth, so keeping a stale copy in the tarball would resurrect old lines.
  const packed = await sandbox.run({
    command: `tar czf ${SANDBOX_ARCHIVE} -C /workspace --exclude=./mindlog.jsonl . && echo ok`,
  });
  if (packed.exitCode !== 0) return `pack failed: ${packed.stderr}`;

  const bytes = await sandbox.readBinaryFile({ path: SANDBOX_ARCHIVE });
  if (bytes === null) return "pack produced nothing";

  await mkdir(dirname(ARCHIVE), { recursive: true });
  await writeFile(ARCHIVE, bytes);
  return `kept ${bytes.byteLength} bytes`;
}

That --exclude is the subtle line in the commit. Without it the mindlog copy rides along in the tarball and gets unpacked at the start of the next session - and then turn.started overwrites it with the real thing, so the stale copy is harmless this time, and quietly wrong the moment anything reads it before the first turn. Cheaper to never store it.

export default defineHook({
  events: {
    async "session.started"(_event, ctx) {
      const sandbox = await ctx.getSandbox();
      const outcome = await restore(sandbox);
      await refreshMindlogCopy(sandbox);
      console.info("[workspace-sync]", ctx.session.id, outcome);
    },
    async "turn.started"(_event, ctx)   { await refreshMindlogCopy(await ctx.getSandbox()); },
    async "session.waiting"(_event, ctx)   { console.info("[workspace-sync]", ctx.session.id, await snapshot(await ctx.getSandbox())); },
    async "session.completed"(_event, ctx) { console.info("[workspace-sync]", ctx.session.id, await snapshot(await ctx.getSandbox())); },
  },
});

Four events, in this commit. The pair of snapshot points was deliberate: session.waiting fired whenever a turn parked, which was the common case and caught work as soon as it was done, while session.completed caught a session that ended without parking - a task-mode heartbeat, for instance.

// ponytail: one tarball, last-write-wins. The chat session and a heartbeat both
// snapshot to the same path, so whoever parks last wins. Per-session tarballs
// or a real file-level sync if that ever costs anything.

Reading its own memory

async function refreshMindlogCopy(sandbox: SandboxSession): Promise<void> {
  let text: string;
  try { text = await readFile(MINDLOG_FILE, "utf8"); } catch { return; }
  await sandbox.writeTextFile({ path: MINDLOG_COPY, content: text });
}

Five lines for gap 2. This commit copied the whole host file at the start of every turn, so grep, jq, wc and any script the agent wrote could work over its own history. It was read-only in effect rather than by permission: appends still went through mindlog_append, so a script editing the copy changed nothing durable. Verified from inside the container - wc -l reported 89 lines, grep -c woke reported 5.

And search

export async function search(query: string, limit = 20): Promise<MindlogEntry[]> {
  const needle = query.toLowerCase();
  const all = await read(Number.MAX_SAFE_INTEGER);
  return all.filter((entry) => entry.text.toLowerCase().includes(needle)).slice(-limit);
}

A linear scan, exposed as mindlog_search. Recall until now was "the most recent N entries", which meant last week was invisible - a real limitation for an agent whose whole point is continuity. The implementation is deliberately dumb: at a few thousand entries a substring scan is instant, and the alternative is an index that has to be maintained and can fall out of sync with the file. The agent used it correctly on the first try, returning five matches for "busywork" and identifying the oldest.

Quiz

Six commits, 2,676 lines, one agent that keeps thinking after you close the tab.