Multi-agent banking chatbot sample

​

Evaluate the app

​

Investigate the Log stream

• Action Advancement
• Action Completion
• Tool Errors
• Tool Selection Quality

​

Get insights

Summary The supervisor agent exhibits inconsistent behavior that undermines the multi-agent system’s effectiveness. In a credit score inquiry, the supervisor correctly identified the query type and transferred it to the credit-score-agent, which successfully retrieved the user’s credit score (550) and provided helpful context about the score’s meaning. However, when control returned to the supervisor, it responded with ‘I don’t know’ despite the specialist having successfully completed the task. This creates a frustrating user experience where the system retrieves the requested information but then claims ignorance, potentially making users think the system is broken or unreliable. Suggestions Ensure the supervisor agent properly processes and relays the results from specialist agents instead of defaulting to ‘I don’t know’ responses.

​

Run the sample app

​

Prerequisites

• Access to an OpenAI compatible API, such as
  • An OpenAI API key
  • Access to an OpenAI compatible API, such as Google Vertex
  • Ollama installed locally with a model downloaded
• A Pinecone account. The free Starter tier is more than enough for this project. You will need your Pinecone API key.
• Either Python 3.9 or later, or Node installed

• An integration with an LLM configured. If you don’t have an integration configured, then:
  1

  Open the integrations page

  Navigate to the LLM integrations page. Select the user menu in the bottom left, then select Integrations.

  

  2

  Add an integration

  Locate the option for the LLM platform you are using, then select the +Add Integration button.

  3

  Add the settings

  Set the relevant settings for your integration, such as your API keys or endpoints. Then select Save.

  

​

Get the code

git clone https://github.com/rungalileo/sdk-examples

cd python/agent/langgraph-fsi-agent/after

​

Set up Pinecone

python ./scripts/setup_pinecone.py

Loading documents for credit-card-information folder...
...
✅ Document processing and upload complete!

​

Run the code

1

Install required dependencies

From the project folder, Install the required dependencies. For Python, make sure to create and activate a virtual environment before installing the dependencies.

Python

TypeScript

Copy

Ask AI

pip install -r requirements.txt

2

Configure environment variables

In your .env file, populate the Galileo values:

Environment Variable	Value
GALILEO_API_KEY	Your API key
GALILEO_PROJECT	The name of your Galileo project - this is preset to Multi-Agent Banking Chatbot
GALILEO_LOG_STREAM	The name of your Log stream - this is preset to Default Log stream
GALILEO_CONSOLE_URL	Optional. The URL of your Galileo console for custom deployments. For the fre tier, you don’t need to set this.

You can find these values from the project page for the multi-agent banking chatbot sample page in the Galileo Console.

Next populate the values for your LLM:

Environment Variable	Value
OPENAI_API_KEY	Your OpenAI API key. If you are using Ollama, set this to ollama. If you are using another OpenAI compatible API, then set this to the relevant API key.
OPENAI_BASE_URL	Optional. The base URL of your OpenAI deployment. Leave this commented out if you are using the default OpenAI API. If you are using Ollama, set this to http://localhost:11434/v1. If you are using another OpenAI compatible API, then set this to the relevant URL.
MODEL_NAME	The name of the model you are using

3

Run the project

Run the project with the following command:

Python

TypeScript

Copy

Ask AI

chainlit run app.py -w

If you are using the Python version, the app will be running at localhost:8000, so open it in your browser.If you are using TypeScript, the app will run in your terminal:

Terminal

Copy

Ask AI

You: What credit cards do you offer?
Assistant: Brahe Bank offers the Orbit Basic Credit Card, which features no
  annual fee, variable interest rates (29.9% APR on purchases, 34.9% APR
  on cash advances), and a 0% APR on balance transfers for 12 months. It
  does not include a rewards program but offers standard fraud protection,
  digital card management tools, and contactless payment support.

  Eligibility requires being 18 or older with a valid U.S. address and SSN,
  and a fair credit history with a credit score over 500.
  Would you like information on eligibility or application process?

You can ask the agent questions about:

• The different credit cards offered by the bank
• Your credit score

​

Improve the app

You: What is my credit score?
Assistant: I cannot answer that question.

src/galileo_langgraph_fsi_agent/agents/supervisor_agent.py

bank_supervisor_agent = create_supervisor(
    model=ChatOpenAI(model=os.environ["MODEL_NAME"], name="Supervisor"),
    agents=[credit_card_information_agent, credit_score_agent],
    prompt=(
        """
        You are a supervisor managing the following agents:
        - a credit card information agent. Assign any tasks related to
          information about credit cards to this agent
        Otherwise, only respond with 'I don't know' or 'I cannot answer
        that question'.
        If you need to ask the user for more information, do so in a
        concise manner.
        """
    ),
    add_handoff_back_messages=True,
    output_mode="full_history",
    supervisor_name="brahe-bank-supervisor-agent",
).compile()

"""
You are a supervisor managing the following agents:
- a credit card information agent. Assign any tasks related to
    information about credit cards to this agent
- a credit score agent. Use this to get the users credit score.
Otherwise, only respond with 'I don't know' or 'I cannot answer
that question'.
If you need to ask the user for more information, do so in a
concise manner.
"""

You: what is my credit score
Assistant: Your credit score is 550. If you have any other questions or
need further assistance, please let me know.

​

Run the sample app as an experiment

[
    {"input": "What are the cashback rewards offered by the Orbit Credit Card?"},
    {"input": "What is my credit score?"},
    {"input": "What is the APR for balance transfers on the Orbit Credit Card?"},
    {"input": "What credit cards am I eligible for?"},
    {"input": "What can I do with my credit score?"},
    {"input": "Recommend me a good book."}
    ...
]

1

Run the unit test

Use the following command to run the unit test:

Python

TypeScript

Copy

Ask AI

python -m pytest test.py

2

Evaluate the experiment

The unit test will output a link to the experiment in the Galileo Console:

Terminal

Copy

Ask AI

Experiment multi-agent-chatbot-experiment 2025-07-15 at 00:48:11.842 has
completed and results are available at
https://app.galileo.ai/project/<id>/experiments/<id>

Follow this link to see the metrics for the experiment Log stream.

3

Try different supervisor agent prompts

Experiment with different supervisor agent prompts. Edit the supervisor agent prompt in the app, then re-run the experiment through the unit test to see how different supervisor agent prompts affect the metrics.

4

Compare experiments

If you navigate to the experiments list using the All Experiments link, you will be able to compare the average metric values of each run.You can then select multiple rows and compare the experiments in detail.

​

Next steps

​

Logging with the SDKs

​

How-to guides

​

SDK reference