References and useful ressources

Traces

Traces are documenting LLM interactions in callback systems called Spans. So each Trace is a collection of Spans collected for a given user interaction with the LLM system (usually via the frontend chat).

Spans

Spans are callback systems that document the interactions with the LLM. They can refer to any unit of execution in the LLM pipeline.

Bellow is an example illustrating one trace (one single interaction). And each unit of execution in the following trace is a span.

traces_spans_01.png|400

Most common span types

Here is an example from the Arize Phoenix debugging tool.
traces_spans_02.png
traces_spans_03.png
traces_spans_04.png

Such tools can generally be used to:


Notebooks from Arize Phoenix

Lab 1

Tracing and Evaluating a LlamaIndex + OpenAI RAG Application
Topics: LlamaIndex, OpenAI, retrieval-augmented generation
Open in Colab Open in GitHub

Lab 2

Tracing and Evaluating a LlamaIndex OpenAI Agent
Topics: LlamaIndex, OpenAI, agents, function calling
Open in Colab Open in GitHub

Lab 3

Tracing and Evaluating a Structured Data Extraction Application with OpenAI Function Calling
Topics: OpenAI, structured data extraction, function calling
Open in Colab Open in GitHub

Lab 4

Tracing and Evaluating a LangChain + OpenAI RAG Application
Topics: LangChain, OpenAI, retrieval-augmented generation
Open in Colab Open in GitHub

Lab 5

Tracing and Evaluating a LangChain Agent
Topics: LangChain, OpenAI, agents, tools, function calling
Open in Colab Open in GitHub

Lab 6

Tracing and Evaluating a LangChain + Vertex AI RAG Application
LangChain, Vertex AI, retrieval-augmented generation
Open in ColabOpen in GitHub

Lab 7

Tracing and Evaluating a LangChain + Google PaLM RAG Application
Topics: LangChain, Google PaLM, retrieval-augmented generation
Open in ColabOpen in GitHub