InfraPrism

Sessions

Sessions let you track costs across multiple LLM calls that belong to the same conversation or workflow. This is essential for chatbots, agents, and any multi-turn interaction.

Why Sessions?

Without sessions, you see individual API calls. With sessions, you can answer:

  • “What’s the average cost per conversation?”
  • “How many turns do conversations typically have?”
  • “Which conversations are most expensive?”

Basic Usage

Add a session_id to group related calls:

import uuid

# Generate a session ID for this conversation
session_id = str(uuid.uuid4())

# First turn
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is Python?"}],
    entity_type="customer",
    entity_id="acme-corp",
    session_id=session_id,
)

# Second turn (same session)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "What is Python?"},
        {"role": "assistant", "content": response.choices[0].message.content},
        {"role": "user", "content": "What are its main features?"},
    ],
    entity_type="customer",
    entity_id="acme-corp",
    session_id=session_id,  # Same session ID
)

Session ID Strategies

Generate a unique ID for each conversation:

import uuid

session_id = str(uuid.uuid4())  # "550e8400-e29b-41d4-a716-446655440000"

Composite ID

Include context in the session ID:

session_id = f"{customer_id}:{conversation_id}"  # "acme-corp:conv-123"

External ID

Use your application’s conversation ID:

session_id = conversation.id  # From your database

Chatbot Example

Here’s a complete chatbot implementation with session tracking:

from infraprism import InfraPrismOpenAI
import uuid

client = InfraPrismOpenAI()

class ChatSession:
    def __init__(self, customer_id: str):
        self.customer_id = customer_id
        self.session_id = str(uuid.uuid4())
        self.messages = []

    def add_system_message(self, content: str):
        self.messages.append({"role": "system", "content": content})

    def chat(self, user_message: str) -> str:
        self.messages.append({"role": "user", "content": user_message})

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=self.messages,
            entity_type="customer",
            entity_id=self.customer_id,
            session_id=self.session_id,
            tags={"feature": "chatbot"},
        )

        assistant_message = response.choices[0].message.content
        self.messages.append({"role": "assistant", "content": assistant_message})

        return assistant_message

# Usage
session = ChatSession("acme-corp")
session.add_system_message("You are a helpful assistant.")

print(session.chat("What is machine learning?"))
print(session.chat("How is it different from AI?"))
print(session.chat("Give me an example."))

Agent Workflows

Track costs across agent tool calls:

def run_agent(task: str, customer_id: str) -> str:
    session_id = str(uuid.uuid4())
    messages = [{"role": "user", "content": task}]

    while True:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            tools=tools,
            entity_type="customer",
            entity_id=customer_id,
            session_id=session_id,
            tags={"feature": "agent"},
        )

        message = response.choices[0].message
        messages.append(message)

        if message.tool_calls:
            # Execute tools and continue
            for tool_call in message.tool_calls:
                result = execute_tool(tool_call)
                messages.append({
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "content": result,
                })
        else:
            # Agent finished
            return message.content

Dashboard Features

With sessions, the dashboard shows:

  • Session list - All sessions with total cost and turn count
  • Session details - Individual calls within a session
  • Session metrics - Average cost per session, average turns
  • Session timeline - Visual representation of conversation flow

Querying Sessions

Via the Analytics API:

# Get session summary
response = requests.get(
    "https://api.infraprism.com/v1/analytics/sessions",
    headers={"Authorization": "Bearer ip-..."},
    params={
        "entity_type": "customer",
        "entity_id": "acme-corp",
        "start_date": "2025-01-01",
    },
)

for session in response.json()["data"]:
    print(f"Session {session['session_id']}")
    print(f"  Turns: {session['turn_count']}")
    print(f"  Cost: ${session['total_cost']:.4f}")

Best Practices

Generate Sessions Early

Create the session ID at the start of the conversation:

# Good - session ID created once
session_id = str(uuid.uuid4())
for turn in conversation:
    response = client.chat.completions.create(..., session_id=session_id)

# Avoid - new session ID each turn
for turn in conversation:
    response = client.chat.completions.create(..., session_id=str(uuid.uuid4()))

Include Session in Logs

Log the session ID for debugging:

import logging

logger = logging.getLogger(__name__)

session_id = str(uuid.uuid4())
logger.info(f"Starting chat session: {session_id}")

Session Timeouts

Consider ending sessions after inactivity:

class ChatSession:
    TIMEOUT_SECONDS = 3600  # 1 hour

    def __init__(self, customer_id: str):
        self.session_id = str(uuid.uuid4())
        self.last_activity = time.time()

    def chat(self, message: str) -> str:
        # Check for timeout
        if time.time() - self.last_activity > self.TIMEOUT_SECONDS:
            self.session_id = str(uuid.uuid4())  # New session

        self.last_activity = time.time()
        # ... rest of chat logic

Next Steps