Chelle AI
Replaced gut-feel prompt engineering with a rubric-based evaluation framework and automated scoring loops.
The problem
Chelle generated study guides and quizzes with LLMs. The team had no systematic way to tell whether a prompt change improved anything, so prompt decisions were made on impressions — someone read a few outputs, liked them, and shipped. That works until two people disagree, and then there is no way to settle it.
What I built
A rubric-based evaluation framework in Python using LangChain against the OpenAI and Gemini APIs, scoring generated material on clarity, accuracy, and topical relevance. With a rubric in place, prompt strategy became something you could run an experiment on rather than argue about: system versus situational prompts, few-shot versus chain-of-thought, scored the same way every time.
Then I wired automated evaluation loops into the FastAPI backend so candidate prompts were re-scored against the test set on every change, instead of depending on someone remembering to spot-check.
The obvious objection
The rubric is scored by a model, which means I was using an LLM to grade an LLM, and the grader has its own biases and its own failure modes. This is a real limitation and I would not present the numbers as though it were not.
What made it useful anyway is that it was consistent. An imperfect judge applied identically to every candidate still ranks candidates correctly far more often than four engineers each reading six samples. The absolute scores I would not defend hard. The ordering they produced held up, and the configurations it selected cut hallucination by 25% and moved engagement 18%.