Encode

Pixels become patches (ViT-class) or edges (Sobel analog here). Language becomes subwords or n-grams. Tables become named dims. Each encoder is named. We do not invent a joint CLIP space on this host.

Attach

Evidence lands on the ticket — files, notes, images, traces. Chat is not the store. A VLM, Kafka hop, or GPU cluster is attachable. Missing never fails the product.

Fuse

Dual-encoder (CLIP/SigLIP-class): cosine in a shared space. Fusion-encoder (LLaVA/BLIP-2/Flamingo-class): a projector maps visual tokens into the LLM. Late fusion: the seated desk combines named encodings. This host fuses late.

Ground

Citations, object support, eval triad. Fluent caption without support is a false complete. A vision-language model, if attached, still goes through the gate. Second family grades.

Dual-encoder

Two towers trained with a contrastive loss (InfoNCE). A vision encoder and a text encoder map into one space. Similarity is cosine. Good for search, zero-shot labels, ranking a caption against a frame. Weak at answering a question about a region with grounded tokens.

Attachable. Unattached analog: 256-d hashing trick + char 3-grams + token bigrams, cosine printed. Not CLIP weights.

Fusion-encoder

A vision encoder emits patch tokens. A projector (MLP or Q-Former) maps them into the language model's token space. The LLM attends to pixels and words together. Flamingo-class uses gated cross-attention; LLaVA-class uses an MLP; BLIP-2-class bottlenecks through a Q-Former. Fluent without object support is a false complete.

Attachable. Unattached analog: Sobel overlay + layer-table parse. Caption goldens sit Trainer / Evaluator. We do not load LLaVA here.

Late fusion at the desk

Modalities do not share a hidden embedding we do not run. Chunk, Sobel, layer table, ABR, files VFS (SHA-256) each encode. The seated playbook fuses. Ground / eval is a second family.

Sobel on Edge overlay. Hash-ngram on Lexical latent + Embedding distance. Chunk strategy named. Missing connectors never fail.

Frozen ViT

The image encoder stays frozen (ViT-g / EVA-CLIP-class). A 224px frame at patch 14 yields (224/14)² + CLS ≈ 257 visual tokens. We do not backprop through those billions of vision params on this host.

Learned queries

A small set of query embeddings — typically 32 — is the bottleneck. They self-attend first so they can specialize (objects, scene text, layout). They are the only visual things the LLM will see.

Cross-attend — the bottleneck

Queries cross-attend into the 257 patches. Information is compressed, not concatenated. Compute is O(queries × patches) in the Q-Former, not O(LLM layers × patches) inside the language model. That is why it is cheaper than Flamingo-class gated xattn.

Two-stage, then project

Stage 1: ITC / ITM / ITG with a text transformer sharing the Q-Former. Stage 2: a linear map sends those 32 embeddings into a frozen LLM as a prefix. Caption fluency is still not object support. Goldens sit Trainer / Evaluator.

Interleave

Frames and words share one sequence: image, text, image, text. Few-shot here means the examples sit in the prompt — in-context, no gradient, no LoRA required to add a visual task.

Perceiver resampler

A set of latent queries cross-attends to a variable-length feature map and emits a fixed token count per image. Extra shots do not explode the window the way raw patches would.

Gated xattn

Cross-attention layers are inserted into a frozen LLM, typically every nth block. A tanh gate is initialized near zero so at step 0 the model is exactly the pretrained language model — vision does not destroy language on day one.

Spend

Every extra shot is another visual token block. Window and dollars grow with shots. spend.halt still applies. Attachable. We do not host Flamingo here.