Claude CodeSoftwareDebugging & incidentsAI score · 72/100AI-assisted edit · no obvious prompt-injection riskX

Build a Prompt Iteration Loop with Langfuse and Claude Code

Input: production Langfuse traces and failure cases → Process: bucket failures into a golden dataset, let Claude Code make the smallest fix → Output: before/after scores and rollback-ready prompt versions

ExperiencedStepsWorkflowTip
Who
Developers or teams running production LLM apps who need systematic prompt iteration
You get
Turn vibe-based prompt editing into traced, scored, rollback-capable product work
You need
Claude Code · Jira · Notion

Constraints

Prerequisites: Langfuse deployed and tracing production calls; Claude Code CLI installed; a ticket system (Jira or Notion) for change records. Boundaries: the author uses 20–50 real failures per golden dataset; rollback if score does not improve; tool brands are swappable, record completeness matters more than the specific tool.

What they did

Nick Papacon replaced intuition-driven prompt iteration with a two-tool loop: Langfuse for tracing and proving problems, Claude Code for making the smallest fix based on trace evidence. The flow has three stages—proof (trace + bucket + dataset), fix loop (smallest change + re-evaluation), and human judgment (approve + rollback + record).

How it went

The author reports this rule is now in active use: a prompt change without a before and after score is just editing; a trace turns it into product work. If the score does not move, roll back. Every change is tied to a specific trace and failure cases. (author_report)

Worth knowing

The source does not specify the exact scoring metric definitions (e.g., accuracy thresholds, cost thresholds), the standard format for the golden dataset, or whether rollback is manual or automated. Teams must define their own scoring criteria. The diagram mentioned in the post is not described in text and cannot be reproduced. The author notes Jira and Notion are interchangeable; tool choice does not affect method validity.

Steps

  1. Instrument production runs with Langfuse

    Integrate the Langfuse SDK into all production prompt calls so every run records: prompt version, model, tokens, cost, latency, tool calls. Confirm traces appear in the Langfuse dashboard.

  2. Bucket failures and build a golden dataset

    Periodically review failed traces in Langfuse and bucket them by failure type: wrong citation, missing context, format break, cost blowout. When a bucket grows, extract 20–50 real failures into a golden dataset (input, expected output, actual output).

  3. Ask Claude Code for the smallest fix and re-evaluate

    Hand Claude Code the Langfuse trace link, failing case IDs, and the golden dataset, using the instruction below to change only the smallest piece. After the fix, rerun the same dataset and compare before/after score, cost delta, and latency delta. No rerun, no ship.

    Execution prompt · Source text。Core instruction in the Claude Code conversation

    change the smallest prompt or harness piece that moves this score

  4. Human approval and rollback decision

    The author approves every shipping change. If the score does not improve, roll back to the prior prompt version immediately. Do not ship prompts without a validated score change.

  5. Record full context in a ticket

    Capture in a Jira or Notion ticket: trace link, failing cases, code diff, before/after scores, and rollback version. The tool matters less than the record.

Prompt you can copy

Setup guidance · Source text。Hand to a setup assistant to confirm inputs and permissions before executing steps

This block is the prompt. Ads are never placed here.

change the smallest prompt or harness piece that moves this score

This prompt is a setup guide, not the runtime instruction inside a Claude Code node. The actual fix instruction is the provenance=source prompt in Step 3.

Source

提示词迭代闭环：Langfuse 追踪失败并分桶成黄金集，把 trace 和失败用例交给 Claude Code 只改最小的一处，前后打分不涨就回滚 · NickPapacon · X · 2026-10-18T16:27:40.000Z

Comments & feedback

Tried this method? Share results, problems or improvements. Comments appear after moderation. Please avoid sensitive information.

Loading…

    Related