Back to Projects
FastAPIPythonRagasMLflowQStashSupabaseGemini 2.5
Istifsar AI Eval Hub
An asynchronous MLOps microservice utilizing FastAPI, Ragas, and QStash to automatically grade RAG responses and track metrics.
Project Overview
A companion microservice to Istifsar AI designed to handle heavy LLM evaluation tasks asynchronously. It uses Upstash QStash to queue webhooks, FastAPI to process them in the background, Ragas to score 'faithfulness' and 'answer relevancy' using Gemini 2.5 Flash, MLflow to log telemetry to DagsHub, and Supabase to sync the final scores back to the user interface.
Role
Backend & MLOps Engineer
- Backend Engineer
- MLOps Engineer
Tech Stack
FastAPIPythonRagasGemini AI APIUpstash QStashMLflowDagsHubSupabaseUvicorn
Backend & Architecture
- Asynchronous webhook queueing via Upstash QStash to prevent frontend blocking.
- Reference-free evaluation pipeline using Ragas to grade faithfulness and relevancy.
- Continuous ML tracking and telemetry logging to DagsHub via MLflow.
- Direct database synchronization to Supabase PostgreSQL using Service Role keys.
App Architecture Diagram
Challenges & Technical Solutions
01—Challenge
Heavy LLM grading taking 30+ seconds caused Vercel frontend timeouts.
↳Solution
Completely decoupled the evaluation into an asynchronous Python microservice using QStash webhooks.
02—Challenge
Free-tier server cold starts causing standard HTTP requests to fail.
↳Solution
Utilized QStash's automatic exponential backoff retries to guarantee delivery even if the server takes 50 seconds to boot up.