Operations specialist · Scale

Data Engineer for AI

Retry storm that double-charges

What this agent actually does

Corpus & feature freshness.

A retry storm that double-charges. Near-dup lines in the corpus nobody counted.

The pain — unique to this desk

A retry storm that double-charges. Near-dup lines in the corpus nobody counted.

The need this desk closes

Exactly-once (uniqueEffects == 1). SHA-256 ingest ledger. Cleaner counts. Drift flag on quality.

Why a generic chat cannot fake this

Retry storm that double-charges

Exactly-once (uniqueEffects == 1). SHA-256 ingest. Near-dups counted before anyone trains.

Takes

Doc dump
Image files as objects

Returns

Ingest note
Exactly-once proof
Drift flag

Modalities this desk actually handles

DocumentsTablesImages as filesStreams analog

How this specialist fuses

Retry storm that double-charges

Files VFS chunks first. Text is extracted for RAG; images stay objects until CV encodes them. Ledger is SHA-256, not a vector claim.

Vision-language on this desk

Named family. Never assumed.

A VLM captioner may be attached to turn image objects into text rows. Unattached, images remain files. No silent caption.

Patch / tokenize → project → fuse → ground is the host mechanism. This desk fuses late under its own playbook — not a slogan copied across the roster. Host mechanism.

A FAQ dump or an event stream needs to be trustworthy. delivery.exactly_once plus broker.publish must keep uniqueEffects == 1. Corpus reindex feeds RAG. caps.drift flags data quality. Corpus cleaner counts duplicate lines, control chars, near-dup Jaccard. No CDC claim. No Debezium. Stale RAG chunks are this desk's incident.

Live missions run in Role Studio. Analog kernels stay in this tab until a hosted call is chosen. Authors never grade themselves. A hashed pack is not a letter. Hire bands are market replacement cost, not EngOS payroll. Isolation is probed, not assumed.

Reports up as Scale. Second family ai-trainer-evaluator. Authors never grade themselves. Isolation is probed, not assumed.

Five analog ticket kinds

How a hired human spends the week. What the analog kernel closes.

Percent splits stay human-week language. Never GPU-loop quotas. Each kind binds to the analog tool that closes it. Escalate only when the kernel cannot.

Data Engineer for AI analog closeHuman bar: Name the ticket. Attach evidence. Second family grades. Analog: #clean. Kernel #/clean on this host. Arithmetic / Canvas 2D. Device none. Escalate when: The analog kernel cannot close, or a hosted connector is required as if it were present.

A typical ticket

What you say. What sits.

Refresh the product FAQ corpus. Prove a retry storm cannot double-charge. Fail the triad if near-dups remain.

Sit keys pipeline / lineage / clean / duplicate. Ticket kinds chore, research, support. RAG Engineer consumes the index. This desk owns uniqueness and freshness.

Playbook

How this agent works the ticket.

01 IngestSources in work context → reindex. Ingest note is the artifact.
02 CleanDuplicate lines, control chars, near-dup Jaccard. Counts only.
03 Exactly-onceRetry storm test. uniqueEffects == 1. Idempotency key is the ticket id.
04 DriftMetric vs baseline on data quality, not only on model score.
05 TriadHard queries after reindex. Fresh chunks, faithful answers.
06 LineageWhat fed the corpus is named. Tribal knowledge is the pain.

Acts like the role

Work it does. Work it will not fake.

Does

Refresh a corpus and verify the RAG triad
Prove exactly-once on a retry storm
Count near-dups before anyone trains
Flag data-quality drift

Does not

Claim Spark / BigQuery / Airflow / Debezium as live here
Train on near-dup lines that were never counted
Let a retry double-charge
Invent CDC

Jobs on the board

Scenarios this desk was built to close.

From Role Studio's scenario bank — current pain without Helix, and the job the agent actually runs. Top eight of twenty.

Corpus / feature freshness

Today. Stale RAG chunks and features

This agent. Ingest pipeline status + reindex job

RAG triad scoring

Today. Fluent answers without faithfulness checks

This agent. Faithfulness + context relevance + answer relevance

Production drift detect

Today. Model gets quietly worse

This agent. Drift monitor on labels/scores + retrain trigger

Golden dataset regression

Today. Prompt/model change ships without suite

This agent. Versioned golden set + offline gate before merge

Cost-aware model routing

Today. Single frontier model burns budget

This agent. Token budget + cascade cheap→frontier

Claim-level citations

Today. Only 50% of sentences supported in audits

This agent. Per-claim support flags + hold if unsupported

Incident runbook agent

Today. Pager chaos without structured triage

This agent. Incident desk with severity + rollback path

Secret / PII exfiltration

Today. Models echo keys from context

This agent. Secret scan on context + artifact write

Skill.md

Reindex corpus

Durable analog execution standard. Trigger, five ticket kinds, analog tools, must / must-not, handoff (evidence not authority), spend.halt, second family. Same factory as PR Creator, Code Reviewer, Handoff Verifier — plus fromDesk(data-engineer-ai).

Trigger. A doc dump must become a trustworthy retrieval source.

01 StepTake the dump
02 StepClean duplicates and control chars
03 StepReindex
04 StepProve uniqueEffects == 1 on a retry
05 StepScore the triad on hard queries

Must

Ingested
Triad pass
uniqueEffects == 1

Must not

Skip the cleaner
Claim a warehouse we did not connect

# Data Engineer for AI

id: data-engineer-ai
layer: scale (Scale)
second family: ai-trainer-evaluator
tools: #/clean

## Pain
The hired Data Engineer for AI seat does not exist yet, or the work has no named ticket.

## Need
A named ticket, an analog close, and a second family. Not a chat that grades itself.

## Isolation
Isolation is probed, not assumed. Unprobed stays unlabeled. Never certified-green from a desk.

## spend.halt
spend.halt on the ticket cap. Only the operator raises the ceiling.

## Ticket kinds
- clean-pass · Data Engineer for AI analog close · analog #/clean · escalate when: The analog kernel cannot close, or a hosted connector is required as if it were present.

Who sits with this desk

Swarm compose by ticket kind.

chore · bar 82
Generative AI Engineer · MLOps Engineer
Low-risk maintenance under standard gates
research · bar 78
AI Research Scientist · Prompt Engineer
Ingest paper, propose capabilities, benchmark, memory lesson

Sit keys

Talk Route matches these words.

clean

Keyword analog in this tab. Not a hosted model. Ticket id is the idempotency key.

RBAC

Pass bar 78. C1.

Ceiling. C1 propose and gated read. Deploy and credential reveal denied.
Deny. credentials.reveal · deploy.canary
Scopes. workspace_read
Swarm seats. chore · research

Daily missions

Run against live engines. Not slides.

delivery.exactly_once · broker.publish

Prove exactly-once for order events

Retry storm must not double-charge

Accept: uniqueEffects == 1. Pain removed: Idempotency keys without tribal knowledge. Surfaces: Systems Lab · Cloud Databases.

Superpowers

SQL engine + dual store
Event bus Avro gate
Exactly-once delivery

Daily jobs

Refresh corpus
Check drift
Validate RAG on hard queries

Skills

ETL
Warehousing
Streaming
Quality

Toolkit

Command Center becomes this desk.

Cloud Storage

Corpus freshness

Sources in work context → reindex

Outcome: Ingest note.

Command Center

Data drift sample

Metric vs baseline

Outcome: Drift flag · caps.drift.

Work templates

ticket

Reindex corpus

Refresh product FAQ corpus and verify RAG triad

Paste: Doc dump. Accept: Ingested · Triad pass.

Surfaces

Nav this desk actually opens.

Command CenterModels & KeysRole StudioCloud StorageCloud DatabasesIntelligence LabEvent AnalyticsCertify & OperateUpdates

Local analog

Corpus cleaner

Duplicate lines, control chars, near-dup Jaccard. Counts only.

Corpus cleaner: duplicate lines, control chars, near-dup Jaccard. Counts only. SAMPLE.

Run Corpus cleaner

Replacement cost

$125,000–$170,000 cash

Year-1 loaded + recruiting $218,300. KORE1 data-engineer analog, AI-tagged.. Not EngOS payroll. Not ARR.

Pilot $0 / Team $79 sits this analog desk. Year-1 hire is $218,300 loaded. That is not a replacement claim. The hired role remains the real thing. The analog desk reports up so one operator can run the ticket.

Open Talk Route All twenty desks

Operations neighbors

Operations specialist

MLOps Engineer

Silent regression that reached production

A silent regression that reached production. Infra canary that ignores answer quality.

Canary only if eval merges. spend.halt armed. Quality SLO, not only infra green.

Release stateProbesVRAM plan

Canary a new prompt/model bundle with rollback criteria
Investigate probe failures
Audit rate limits and breakers

Analog · VRAM allocator · bar 82 · C2

Operations specialist

AI Infrastructure Engineer

Finance finds the runaway agent first

Finance finding the runaway agent first. A cluster claim with no GPU, batch, or seq.

Token bucket → breaker → fallback. Cluster analog you set. Cost ledger vs budget.

ServingCostAttachable cluster

Tune model routes and watch the cost budget
Contain a runaway agent with the enforce chain
Canary an inference build

Analog · Cluster bubble sandbox · bar 82 · C2

Data Engineer for AI

Corpus & feature freshness.

A retry storm that double-charges. Near-dup lines in the corpus nobody counted.

Exactly-once (uniqueEffects == 1). SHA-256 ingest ledger. Cleaner counts. Drift flag on quality.

Retry storm that double-charges

Retry storm that double-charges

Named family. Never assumed.

How a hired human spends the week. What the analog kernel closes.

What you say. What sits.

How this agent works the ticket.

Work it does. Work it will not fake.

Scenarios this desk was built to close.

Corpus / feature freshness

RAG triad scoring

Production drift detect

Golden dataset regression

Cost-aware model routing

Claim-level citations

Incident runbook agent

Secret / PII exfiltration

Reindex corpus

Swarm compose by ticket kind.

Generative AI Engineer · MLOps Engineer

AI Research Scientist · Prompt Engineer

Talk Route matches these words.

Pass bar 78. C1.

Run against live engines. Not slides.

Prove exactly-once for order events

Command Center becomes this desk.

Corpus freshness

Data drift sample

Reindex corpus

Nav this desk actually opens.

Corpus cleaner

$125,000–$170,000 cash

MLOps Engineer

AI Infrastructure Engineer

EngOS