Memory Management Guide

Memories are the fundamental unit of information in TokST. Memories are tagged for filtering, organized within an atlas, and embedded into a 1536-dimensional vector space when the configured embedding provider is available.

Memory Types

Choose the type that best describes the information you're storing.

tokst remember "The database runs on Supabase Postgres" --type fact
tokst remember "Use Turborepo for monorepo management" --type decision --tags architecture,infra

Write structured Markdown

Plain text works well for short facts. Use Markdown for decisions, architecture, meeting notes, and tasks so each memory remains easy to scan in the dashboard.

The dashboard editor offers these templates and Markdown formatting controls. For long CLI input, keep the content in a Markdown file:

tokst remember --type decision --tags api,auth --stdin < decision.md

Agents capture confirmed durable information in the same structure. Keep credentials, private keys, raw reasoning, and transient tool output outside memory records.

Kind

Every memory includes a kind field that describes its derivation:

The kind is assigned automatically but can be overridden.

Source Types

Track where a memory originated:

tokst remember "Auto-scaling group configured for 2-10 instances" --source-type agent --source codex

Tags

Tags are comma-separated labels used for filtering and discovery. Unlike types (which are mutually exclusive), tags are additive — a memory can have many tags.

tokst remember "Deploy process documented in Notion" --tags deploy,documentation,notion

Tags power filtered searches:

tokst search "deploy" --tags production
tokst search "architecture" --type decision

Search

TokST uses a keyword-first, semantic-fallback search pipeline:

1. Keyword matching — Traditional text search on memory content
2. Semantic vector search — Embedding similarity in 1536-dimensional space

Direct keyword matches return immediately and avoid an embedding round trip. When no keyword result exists, TokST generates a query embedding and runs a 1536-dimensional vector search constrained to the authenticated user and accessible workspaces.

tokst search "database connection issues"       # Semantic
tokst search "Supabase connection string"       # Keyword match
tokst search "auth" --type architecture         # Filtered
tokst search "api" --limit 20 --json            # With options

Context Snapshots

The context command generates a formatted snapshot of recent memories in the active atlas. This is particularly useful for providing conversation context to AI agents.

tokst context                         # Active atlas
tokst context --atlas <id>            # Specific atlas
tokst context --limit 50              # More memories

The output includes memory content, type, tags, and timestamps in a readable format designed to be consumed by both humans and agents.

Memory Lifecycle

Memories progress through three states:

Active  -->  Archived  -->  Deleted

tokst memory archive <id>     # Soft-hide
tokst memory restore <id>     # Bring back
tokst memory delete <id>      # Permanent

Trusted Memory Lifecycle

Use trusted metadata for facts, decisions, and policies that need a clear source or review trail.

tokst memory verify mem_xxx --evidence https://example.com/policy --confidence 0.95
tokst memory verify mem_xxx --valid-until 2027-01-01T00:00:00Z
tokst memory supersede mem_old mem_new

Verification preserves the memory and marks it as reviewed. Superseding links an older memory to its replacement and marks the earlier record as superseded, keeping historical context available.

Embedding

When a memory is stored or its content changes, TokST requests an embedding vector. The embedding process:

• Produces a 1536-dimensional vector when the provider is configured and available
• Runs on the server at write time
• Is transparent — you never interact with vectors directly
• Powers semantic fallback queries

Memory writes remain available when the embedding provider is not configured; keyword search continues to work and semantic fallback becomes available after embeddings are generated.

File Attachments

Memories can have files attached via the --file flag. See the File Attachments Guide for details.

tokst remember "Sprint planning notes" --file sprint-planning.pdf
tokst memory attach <id> --file diagram.png

Batch Import

Import an entire folder of files as memories. Text files (md, code, json, csv, etc.) are auto-extracted; binary files are uploaded as attachments.

tokst import ./docs --dry-run                  # preview
tokst import ./docs --type note --tags imported # import

See the CLI Reference for all flags.

Best Practices

• Use types consistently — This makes filtered searches more reliable
• Tag liberally — Tags are the primary mechanism for cross-cutting organization
• Archive, don't delete — Archived memories can be restored if needed
• Take context snapshots — Run tokst context before agent sessions to provide background
• Append to existing memories — Use append instead of creating duplicates when adding related information