AI Test Automation

This feature enables you to efficiently test one or several configurations of your generative AI setup.

Instead of manually typing individual questions into the chat, you can upload a list of test questions (as a CSV or XLSX file). For each question, an answer will be automatically generated by the AI process as configured in your process model. This includes the AI Gateway, all AI knowledge sources, and other relevant modules. After a processing time (typically around 30 minutes, depending on the number of questions), you can download the generated answers as a CSV file. This allows you to easily evaluate your AI's performance, especially after making changes to its configuration (e.g., updated modules, prompts, system messages, knowledge sources, or LLMs).

Since the AI tests are specific to the process, you can only test the AI of a particular process/agent. To upload test questions and test the AI, navigate to the Knowledge tab of the process and click on the AI Test Automation button.

Upload file format

The file for uploading questions should be a CSV or XLSX file with the following columns:

• question: The question that should be answered by the AI.
• locale: The locale (language) in which the question should be answered. This is important for the language model to generate the correct answer. The locale should be in the format en for English, de for German, etc.
  • Please note: Only locales that are configured and active in the agent (process) are supported.
• conversation (optional): The name of a conversation in which the question is answered. Questions with the same conversation name are evaluated as consecutive turns of one conversation, so that each answer is generated with the previous turns as context. Leave this column empty to evaluate a question in isolation.

The questions are processed from top to bottom in the order of the file. This matters for the conversation column: the turn order of a conversation follows the row order, so place the turns of a conversation in the file in the order in which they should be answered.

info

The column was previously called dialog. Upload files that still use the old dialog column are rejected; rename the column to conversation before uploading.

Example file content:

question	locale	conversation
How do I cancel my subscription?	en	cancel
And can I get a refund?	en	cancel
What is the capital of France?	en
Suche ein Produkt mit einer Bewertung über 4 Sternen.	de