In this blog we will understand process of utilizing the Monitoring and Evaluation (METRO) features of Fusion AI Agent Studio. METRO capabilities help monitor performance of the AI agents. We can also evaluate the accuracy of the agents as well as track the interactions with the agents and identify the errors.
AI Agent Studio - Monitoring
Monitoring Dashboard Prerequisites
In order to get the monitoring dashboard working for insights on performance, cost and reliability, we first need to run (or schedule) the ESS job 'Aggregate AI Agent Usage and Metrics'.
- Let's go to Tools -> Scheduled Processes
- Search for ESS Job 'Aggregate AI Agent Usage and Metrics' and submit it. This job doesn't have any parameters.
- We can also schedule this process to run on a set schedule (say every 8 hours or so).
- Confirm that the ESS job is successfully completed:
Monitoring dashboard
- Navigate back to AI Agent Studio - > Monitoring and Evaluation
- Click on Monitoring tab
- We should now see all the details on the Monitoring dashboard.
- It shows performance metrics such as Error rate, Error count, Sessions, Latency etc.
If we open a specific agent from the list, we will be able to see all the sessions as well as agent's error rate, error count, total tokens consumed etc.
- If we navigate to Value tab, it will show us an approximate aggregate value of how many hours (and cost) these AI Agents have saved for us. It will also show the time saved by each individual agent.
- In this case, if we open the "1 Agent Teams" link under Recruiting, we will see our custom agent's name as well as the value metrics for the same.
AI Agent Studio - Evaluation
Evaluation sets are useful in analyzing the performance of our AI Agents. A given evaluation set may have one or more test cases/questions, their expected responses as well as the metrics to be measured. The evaluation sets are specific to each agent. We can create multiple evaluation sets for a given AI Agent.
Let's understand the steps involved in creating the evaluation sets, executing the evaluations and understanding the results.
Creating Evaluation Sets
- Click Manage Evaluations and select Add (+)
- Enter a name/code, description for the evaluation set. Select the agent team we want to evaluate.
- Choose the run mode (Sequential or Random). This may differ based on your use case.
- Sequential: This mode runs the questions in the exact order you define them. It's recommended to use this if one question depends on the context of the previous one.
- Random: This mode runs the questions in a random order.
- From the Questions tab, add the desired questions that would be most likely asked to the AI the agent. Also, enter the expected answers that the agent should deliver. Make sure that the questions and answers are concise. You can add the questions / answers manually or upload a CSV file with the questions in the column 1 and the expected answers in the column 2.
- Navigate to the Metrics tab. Here we can edit each metric to set the pass and fail criteria. For example, to indicate that the test fails if the correctness score is less than 0.5, choose < as the threshold condition and enter 0.5 as the threshold value.
- Click Create to save the evaluation set.
Running Evaluation Sets
Navigate to the Manage Evaluations page, select the Initiate Evaluation Run action for the desired evaluation set.
- Let's choose the Draft version of the agent team and click Run.
- We should see a confirmation message.
- To monitor the evaluation run, go to View Evaluation Runs action
- We should see the run in progress.
Analyzing the evaluation run results
- Once the Evaluation Run is completed, we can view the summary results
- To see the detailed results, we can navigate to View Run Results option.
- We can see details on Response performance as well as Correctness of the Agent.
Response performance: This shows the comparison of the expected response versus the actual response from the agent for each question, along with the metrics for each question in the evaluation.
Trace: This provides information about the detailed timeline for each question in the evaluation.
Correctness: This gives us a detailed breakdown of the correctness score. The LLM provides an initial score and feedback.
We can add our own feedback for record keeping in the Correctness Score by Human column.
Comparing Evaluation Runs
We can see the comparison of two different evaluation runs (for the same evaluation set), and understand the regressions or improvements in latency, correctness, and token usage.
- Select any two evaluation runs and click Compare.
- The Summary tab displays a high-level overview of the performance differences between the runs.
- The Details tab provides a granular breakdown of the runs. We can directly compare the actual response from Run 1 against the actual response from Run 2 for each given question.
We can also compare the other metrics such as latency, tokens used, and trace links for each question.
This helps us in understanding changes in the performance as well as accuracy of our AI Agent.
This is how we can utilize the METRO functionality of Oracle Fusion AI Agent Studio to monitor the AI Agent's performance as well as evaluate the agent's response, correctness, and other metrics such as latency, tokens consumed and errors.
This is very useful in making sure that the AI Agent is functioning as expected before we deploy it in the production environment.