Generation Logs & Cost Tracking

Every LLM call in Pipelines is logged with the prompt, response, token counts, cost, and latency. Per-task detail lives inside the Data Explorer task detail view, and pipeline-wide aggregates live on the LLM Analytics tab.

What is logged

Each LLM generation attempt writes one row to llm_response_logs. Key fields include:

Where to see the logs

Per-generation (task detail)

Open any task from the Data Explorer task table. Once an LLM field has generated, it shows a green left border accent with a summary line displaying Model · N tokens · $cost. Before generation, the field shows a placeholder ("LLM response will be generated") with no log details.

Click that summary to expand it and see:

• The resolved prompt sent to the model (after variable substitution).
• The system prompt, if one was configured.

A Thinking button (purple, with a brain icon) appears inline in the summary when the model produced internal-reasoning blocks — clicking it opens a dialog displaying the thinking content.

A Tool Calls (N) collapsible section appears below the field, loaded lazily. Each entry shows the tool name, arguments, result, latency, any error, and which round of the tool-calling loop it belonged to.

If the field has been regenerated more than once, ‹ / › arrows appear next to the field label so you can page through previous generations. The model, tokens, cost, thinking, and tool-call trace all update to reflect the selected generation.

In-progress generation (LLM Pending badge)

While an LLM field is still generating, an amber LLM badge with a spinner appears in the task's status column. Clicking it opens a live progress dialog showing:

• Which field and model is currently generating.
• Streaming output text as it arrives.
• Tool call status (e.g. "Calling tool search_web").
• A count of fields and nodes completed so far.

The dialog polls automatically. When generation finishes, the parent table refreshes — close the dialog to see the updated results.

Failed generation (LLM Failed badge)

If an LLM call fails after all retries, a red LLM Failed badge appears in the task's status column. Clicking it opens a dialog with the field name, model, attempt number, timestamp, and the raw error message.

Pipeline-wide (LLM Analytics tab)

In the Data Explorer, switch to the LLM Analytics tab. It aggregates every generation in the pipeline and shows:

• Summary strip — total calls, success rate, total tokens, total cost (USD), and average latency.
• Cost Over Time (or Token Usage Over Time when no pricing data is available) — a daily time series chart.
• Calls by Model — bar chart of every model that produced at least one generation.
• Calls by Field — bar chart of the top 10 most-called fields.
• Field Details table — collapsed by default. Expand it to see a per-field breakdown with Model, Calls, Tokens, Cost, Success rate, Avg latency, and a brain icon marker for fields that received thinking traces. All columns are sortable.

The task table itself does not show token, cost, or latency columns. For LLM fields, the task table shows the generated value alongside a small model pill (a chip with a CPU icon and the model name) — drill into the task detail or the LLM Analytics tab for full numbers.

Tool call logs

When an LLM uses tool calling, every round of the tool loop is logged to llm_tool_call_logs and surfaced in the Tool Calls collapsible section on each LLM field. Each entry captures:

Cost tracking

Cost is estimated per generation from per-token pricing:

• Platform models — cost is computed using LiteLLM's built-in pricing table. If LiteLLM cannot price the model, the platform falls back to reading the cost from the provider's response (e.g. OpenRouter's usage.cost field). If neither source is available, cost_usd is stored as null.
• Custom (BYOK) models — pricing is optional. The Add Custom Model dialog has Input cost/token (USD) and Output cost/token (USD) fields. If you leave them blank, generations from that model are logged without a cost value, and the LLM Analytics chart falls back to showing token usage instead of dollars.

Costs are aggregated at:

• Per generation — visible in the LLM details summary for each field.
• Per pipeline — the totals on the LLM Analytics tab.

There is currently no organization-wide cost rollup surface — each pipeline has its own LLM Analytics tab.

Error handling

When an LLM call fails (timeout, rate limit, provider error, invalid response):

1. The attempt is written to llm_response_logs with success=false and the error message.
2. The platform automatically retries transient failures with exponential backoff (delays of 1s, 2s, 4s). The retry count is controlled by the LLM_MAX_RETRIES server environment variable (default 3 retries, so up to 4 total attempts). This is not exposed as a per-pipeline setting.
3. If all retries fail, the field shows an error state and the task gains an LLM Failed badge in Data Explorer.
4. If the node has Allow regeneration enabled (llmRetriable), the contributor can click Regenerate on the field. This is bounded by the Max retries setting on the node (llmMaxRetries, 1–10) — this caps how many times a user can regenerate, not the automatic retry count.

Debugging checklist

1. No cost showing for a generation. Check whether the model has pricing configured. For BYOK models, open Add Custom Model and fill in the cost-per-token fields.
2. LLM Analytics tab missing. The tab only appears when the workflow has at least one LLM-generated field. Confirm you have an llm_response field on a node.
3. Tool calls not appearing. The Tool Calls section loads lazily — it polls every 5 seconds for up to 60 seconds. Expand the LLM details panel and wait a moment.
4. Generation arrows not showing. The ‹/› navigation only appears when a field has 2 or more generations. If the field was only generated once, no arrows are shown.
5. LLM Failed badge but no error detail. Click the badge to open the error dialog. Check the attempt number — if it equals 4 (default), all automatic retries were exhausted.