Vision Doc — Agent J v1.0 · April 2026

"What architectural features distinguish early Pallava temples from imperial Pallava temples, and which scholars agree vs disagree on the transition period?"

Without agent: One RAG query, top 5 chunks, one answer. Likely incomplete — a single query cannot capture both periods and the scholarly debate simultaneously.

With agent: Claude queries "early Pallava architecture", then "imperial Pallava architecture", then "Pallava temple transition period", compares Minakshi vs Sastri vs Longhurst on the dating, surfaces the disagreement, and synthesises a complete answer — all in one instruction.

Prerequisite: Tag filter + source weighting (Phase 2) must be in place for the agent to isolate and compare sources meaningfully.

"Translate this Pallava Grantha inscription fragment: śrī-nara…"

Without agent: One-shot prompt with whatever context is retrieved. If a character is ambiguous, the translation is a guess with no transparency.

With agent: Claude attempts the translation, identifies the ambiguous character, queries the script training registry for that specific glyph, retrieves the confidence score and Unicode mapping, retries the translation with that grounding, and explicitly flags any remaining uncertainty.

Prerequisite: Script registry API exposed as an agent tool (GET /training/image/file/{filename} already exists — needs a tool definition wrapper).

"How did Pallava temple iconography influence early Chola temples at Thanjavur?"

Without agent: Searches one collection, returns Pallava chunks OR Chola chunks — rarely both in the right proportion for a meaningful comparison.

With agent: Queries Pallava sources first, then Chola sources, identifies overlapping iconographic elements mentioned in both, and constructs a comparative answer with citations from each tradition.

Prerequisite: Consistent tagging convention across all ingested sources. If Chola sources are not tagged chola, the agent cannot isolate them. Tag discipline during ingestion is critical.

"Write a 2,000-word research note on Mahendravarman I's contribution to rock-cut architecture"

Without agent: Single RAG call, limited context, generic structure. Cannot accumulate evidence across multiple retrieval steps.

With agent: Queries the KB section by section (early reign → cave temples → inscriptions → scholarly dating debates), accumulates evidence across multiple retrieval steps, resolves conflicts between sources, then generates a structured paper with inline citations — and delivers to Word/PDF when done.

Prerequisite: A structured system prompt defining output format, section headings, citation style, and the ideological guardrail. This is the most critical single instruction for the agent.

"What do we know about Pallava naval history?"

Without agent: Returns whatever 5 chunks match, even if corpus coverage is thin. No signal that the answer is incomplete or unreliable.

With agent: Queries the topic, counts the evidence, and explicitly responds: "Only 2 chunks found across 3 sources — corpus coverage on Pallava naval history is limited. The available evidence suggests… but this should be treated as preliminary until more sources are ingested."

Prerequisite: Confidence reporting instruction in the system prompt — Claude must be told to always state source count and flag low-coverage topics explicitly.

What works in your favour: The pluggable ingester pattern (BaseIngester → PdfIngester, DocxIngester, etc.) is clean and extensible. The RAG pipeline (rag/answerer.py + rag/retriever.py) consists of two separate, independently callable functions — a History Agent can wrap them without modification. Claude's native tool-use is the correct routing mechanism. The FastAPI router pattern (app/routers/) means adding an agent.py router is a natural extension.

The caveat — client coupling: app/dependencies.py returns a hardcoded anthropic.Anthropic() client. When the Master Agent needs to call sub-agents with different models (Haiku for fast routing, Opus for deep research), there is no way to configure this without touching multiple files. Fix: replace hardcoded model strings with os.environ.get("MODEL_SMART", "claude-opus-4-6") before wiring the agent loop.

The bigger risk is state: Background jobs (_jobs dict in admin.py) are in-memory. When a Master Agent starts a long-running research paper task, the result disappears on server restart. For the History Agent alone this is acceptable. Personal Agent (reminders, goals, Hindu calendar, Gmail across sessions) requires a persistent SQLite store from day one.

Summary: Flexible for History Agent now. Personal Agent needs SQLite state persistence first. Security Agent can run as a stateless audit loop.

Claude Code: Yes, immediately, no code changes needed. Claude Code's MCP tool interface can call any HTTP endpoint. You already have POST /agent/run planned. Write an MCP tool definition that calls this endpoint and Claude Code becomes a natural language front-end to your entire knowledge base today. This is the fastest path to a working agent interface.

Apple app (iPhone): Yes, but it is a separate project. Your FastAPI backend is REST-over-HTTP. Any iOS app can call it. The architecture is correct — app/main.py is already described as "the backend for a future iOS/Android app." What you need to build: a SwiftUI app with a chat-style UI, auth (a fixed API key in the app is sufficient for personal use), and the /agent/run endpoint on the backend. The iOS app does not change any backend design decisions. Build the backend first; the app is a client.

One practical issue with both: Long-running requests. Research paper generation takes 15–30 seconds. The fix is the same for both interfaces: convert long-running agent tasks to background jobs with polling (POST /agent/run → returns job_id → GET /agent/job/{id} polls for result). This pattern already exists in admin.py for PDF ingestion — apply it to the agent loop in the same way.

Issue 1 — Local file paths (most significant): source_library.py copies files to C:\Users\siddi\OneDrive\Personal\History\Pallavas\source_library\. On EC2 there is no OneDrive. Since this is already controlled by the PALLAVA_LIBRARY_DIR env var, this is a configuration change not a code change — but you need to plan where files live on EC2 (e.g. an EBS volume at /data/source_library) before migrating.

Issue 2 — ChromaDB is local (manageable): ChromaDB's PersistentClient writes to a local directory. On EC2 this directory must be on a persistent EBS volume, not the ephemeral root volume. Standard deployment practice — no code change required.

Issue 3 — docx2pdf is Windows-only (blocking): On Windows, docx2pdf uses Microsoft Word via COM automation. On Linux EC2 there is no Word. Replace with LibreOffice (libreoffice --headless --convert-to pdf) or Gotenberg. This is a 10-line code change but it will break on first deploy to Linux EC2 if not addressed.

Issue 4 — py -3 command: On EC2 (Linux), the Python command is python3, not py -3. Any shell scripts or subprocess calls using py -3 will fail. Minor — worth noting for deployment scripts.

Issue 5 — Unicode fonts for Word export (minor): The .docx research paper export uses Noto Serif to render IAST diacritical characters (ā, ī, ū, ṭ, ḍ etc.). On Windows this font must be installed manually. On EC2 (Ubuntu/Debian), run: sudo apt-get install fonts-noto-serif as part of the server setup script — one line, resolves all Unicode rendering issues in exported Word documents.

No structural issues. FastAPI + uvicorn is production-grade and runs identically on EC2. Tighten CORS (allow_origins=["*"]) before going public.

The numbers: 1,000 PDFs × ~100 chunks per PDF = ~100,000 chunks. 10,000 images × 1 description = 10,000 vectors. Total: ~110,000 vectors at 1024 dimensions. ChromaDB's HNSW index handles 10M+ vectors — 110K is trivial. At 100K vectors, HNSW queries return in <100ms locally. On EC2 with an EBS volume expect 200–400ms. Fully acceptable.

The real scaling risk is not ChromaDB — it is knowledge.json: knowledge_store.py loads the full knowledge.json into memory on every ingest operation. At 22.7 MB today (~5,000 chunks) this is fine. At 100,000 chunks this file will be ~450 MB and loading it on every ingest will be slow. The fix: stop using knowledge.json as a query store and query ChromaDB directly for everything. knowledge.json becomes an append-only audit log, not a lookup table. No re-ingestion needed — this is a code change to the ingest path only.

Multi-dynasty collection design — recommended approach: Rather than one collection per dynasty (chalukya_knowledge, chola_knowledge), use a single indian_knowledge collection with a dynasty metadata tag. All dynasties in one collection, filter by tag at query time. This makes cross-dynasty queries simple and natural — exactly what is needed for "cross-dynasty relationship details without hallucination." Migrate to this model when Chalukya ingestion begins — it is a reindex operation, not a code change.

Why Claude wins for this project specifically: The critical advantage is contextual understanding. When Claude sees a damaged inscription photograph with partial characters, it can reason: "This is a Grantha script Pallava copper plate, therefore this partially visible character is likely..." — Textract and Document AI do pixel-level pattern matching, not contextual reasoning. For scholarly epigraphic work this is not a small difference — it is the entire problem domain.

Where competitors are cheaper: For structured documents (invoices, forms, tables), Textract's AnalyzeDocument API is faster and cheaper. For high-volume modern printed text, Google Document AI is cheaper at scale. Neither of these scenarios applies to the primary use case.

Recommendation: Keep Claude for OCR. The OCR evaluation suggested in the requirements is worth doing for modern Indic printed text where Google's dedicated Indic models might reduce cost at scale. For inscription photographs and rare scripts, Claude is in a category of its own.

ChromaDB API calls appear in four files. The APIs between providers are not compatible — Pinecone, Weaviate, Qdrant, pgvector all have different client libraries and query syntax.

Pragmatic approach (recommended): Before going live on EC2, create a vector_store_factory.py with a thin VectorStore wrapper class that encapsulates the 4 operations actually used (upsert, query, delete, count). All 4 files import from this factory. When migrating, change one file. This is 2–3 hours of refactoring that buys clean migration later.

Hosted alternatives for Phase 4: Qdrant Cloud — closest API to ChromaDB, easiest migration. pgvector (PostgreSQL extension) — if one database for everything (agent state + vectors) is desired, compelling for EC2. Pinecone — managed, no EC2 required, but vendor lock-in and higher cost at scale.

Priority: Fix the synchronous blocking issue before launching the agent loop — a 60-second HTTP request will time out on mobile and feels broken. Background jobs + polling solves this and is already a proven pattern in the codebase.