Memory Pipeline

Every piece of data that enters SharedMemory passes through a multi-stage pipeline.

Pipeline Stages

Input → Classify → Embed → Guard → Index → Extract → Broadcast

1. Classification

The message classifier determines if the input contains indexable knowledge. Casual conversation is skipped.

2. Embedding

Text is embedded using all-MiniLM-L6-v2 (384 dimensions). Embeddings are cached in Redis for 30 days.

3. Guard System

Score Range	Action	Reason
< 0.55	Auto-approve	Low similarity — clearly new information
0.55 – 0.97	LLM Guard	Ambiguous — LLM evaluates conflict
> 0.97	Auto-reject	Near-duplicate detected

4. Indexing

Approved entries are indexed across three stores: Qdrant (vector), Neo4j (graph), Postgres (record).

5. Knowledge Extraction

Extracts entities, facts, and relationships. Generates/updates entity summaries.

6. Broadcasting

Real-time WebSocket notifications inform connected clients.

Async Processing (BullMQ)

Background memory ingestion uses BullMQ backed by Redis:

• Jobs persist across API restarts and pod cycling
• 3 retry attempts with exponential backoff
• Concurrency: 5 workers for propose queue, 3 for guard queue
• Failed jobs retained for debugging (up to 500)

Document Ingestion

1. Parse — PDF, DOCX, TXT, MD, CSV, JSON extracted to text
2. Chunk — Split into overlapping chunks
3. Store — Chunks saved to Postgres
4. Embed + Index — Each chunk embedded and indexed in Qdrant
5. Extract — Knowledge extraction runs on full document text
6. Graph Link — Document node linked to extracted entities in Neo4j

Supported Formats

Format	Extension	Max Size
PDF	.pdf	20 MB
Word	.docx	20 MB
Plain Text	.txt	20 MB
Markdown	.md	20 MB
CSV	.csv	20 MB
JSON	.json	20 MB