Optimizing GPT-4 Prompt Engineering
Prompt design and evaluation for virtual support and AI assistant performance.
Key Details
| Challenge | A support assistant drifted, hallucinated policy and had no eval harness. |
|---|---|
| Solution | Prompt packs, golden sets and regression tests before every prompt change. |
| Technologies | GPT-4, LangSmith, pytest, FastAPI |
Technologies used
OpenAI
Python
FastAPI
pytest
Docker
Client background
A customer-support product shipped an LLM assistant that looked good in demos but drifted in production. Policy answers hallucinated, tone varied by prompt tweak, and nobody could prove a change was safer before release.
Key challenges
- Prompt changes went live without regression coverage against real tickets.
- Hallucinated policy replies created escalations and trust issues with agents.
- No shared golden set — quality debates were anecdotal.
- Multiple teams edited prompts in isolation, causing conflicting behavior.
What we built
- Versioned prompt packs with ownership and changelog discipline.
- Golden evaluation sets drawn from real support transcripts.
- LangSmith + pytest regression gate before every prompt or model change.
- FastAPI serving layer with structured logging for failure review.
Project team: 4 engineers across AI/ML, backend and domain specialists — delivery over 10 weeks.
How we delivered
01
Baseline
Captured failure modes from production logs and agent escalations.
02
Golden set
Built labeled examples for policy, tone and tool-use correctness.
03
Harness
Wired evals into CI so regressions block release.
04
Operate
Defined a change process for prompt owners and support leads.
Business impact
MeasurableAnswer quality
SaferPolicy replies
RepeatableReleases