Overview
A self-hosted RAG observability stack that decomposes LLM responses into atomic claims, attributes each claim to its source chunk via cosine similarity, and runs an LLM-as-judge faithfulness check. It surfaces a per-claim verdict — faithful, partial, unfaithful, or refusal — instead of a single opaque faithfulness score.
The Problem
Most existing eval tools (RAGAS, LangSmith, Braintrust) collapse two fundamentally different failure modes, bad retrieval and bad generation, into one faithfulness number. A team debugging a wrong answer has no way to know from that score alone whether the retriever pulled the wrong chunk or the model hallucinated on top of a correct one. ContextLens exists to make that distinction explicit, at the level of individual claims within a response, not the response as a whole.
Architecture / How It Works
FastAPI backend, Celery workers for async processing, PostgreSQL with pgvector for embedding storage and similarity search, Redis for task queuing and caching, a Next.js frontend for the dashboard, and a lightweight Python SDK that instruments the calling application. The SDK's context manager captures a trace (query, retrieved chunks, generated response) and ships it to the backend on a background daemon thread, fire-and-forget, so instrumentation never blocks the caller's actual request path.
Key Decisions
- IVFFlat over HNSW for the pgvector index: for a self-hosted, single-developer threat model at realistic dataset sizes, IVFFlat's simpler build and lower memory footprint won out over HNSW's marginal recall improvement.
- asyncpg with raw SQL instead of an ORM: full control over query shape and performance for pgvector similarity queries.
- Fixed-window over sliding-window rate limiting: matched the actual single-user threat model instead of over-engineering for scale that doesn't exist yet.
- Quote-first reasoning in the LLM judge prompt: the judge quotes the exact span of the source chunk it relies on before rendering a verdict, grounding judgment in specific text rather than abstract reasoning.
- Frozen-response retest methodology: when fixing attribution or judging logic, the same frozen set of previously generated responses is retested against new logic, isolating whether a fix actually improved classification accuracy.
Challenges & Solutions
The attribution threshold was calibrated against an organic 14-query RAG test run, not synthetic data. The initial 0.75 cosine similarity cutoff misclassified paraphrased claims as retrieval failures, since paraphrasing lowers surface-level similarity even when meaning is preserved. Fixed with a three-band confidence model: a clear match above 0.75, a clear non-match below 0.65, and a low-confidence band from 0.65 to 0.75 escalated to the LLM judge instead of auto-classified. Separately, refusal detection was added so a model correctly declining to answer isn't counted as a hallucination.
Tech Stack