Setup
In this example, we’ll use thegpt-4o
model to generate a response to for a simple prompt. We’ll enable the instruction_adherence
metric to monitor the model’s adherence to the prompt.
To calculate metrics, you will need to configure an integration with an LLM. Visit the relevant API platform to obtain an API key, then add it using the integrations page in the Galileo Console.
What showed up in metrics:
When we examine the Instruction Adherence metric, we see a score of 0.6667:
Metric Explanation
The instruction provided was to 'Explain the following topic succinctly: Newton's first law'. The response begins by defining Newton's First Law and provides a clear explanation of the concept of inertia. However, the response is lengthy and provides more detail than the word 'succinctly' implies. While it does effectively cover the essence of the topic, it could be more concise to align better with the instruction. Thus, while informative, the response does not fully adhere to the request for a succinct explanation.
What went wrong?
As the explanation indicates, the main reason the model did not follow the instructions is that the prompt is too vague. The prompt does not provide enough information about what constitutes a “succinct” explanation.The solution
In order to fix this, we can modify the prompt to provide more specific instructions: