Grader

String Grader

Simple pass or fail grader that returns 0 or 1.

• eq = Returns 1 if the input matches the reference (case-sensitive), 0 otherwise
• neq = Returns 1 if the input does not match the reference (case-sensitive), 0 otherwise
• like = Returns 1 if the input contains the reference (case-sensitive), 0 otherwise
• ilike = Returns 1 if the input contains the reference (not case-sensitive), 0 otherwise

{
    "type": "string_check",
    "name": string,
    "operation": "eq" | "ne" | "like" | "ilike",
    "input": string,
    "reference": string,
}

Text Similarity Grader

{
    "type": "text_similarity",
    "name": string,
    "input": string,
    "reference": string,
    "pass_threshold": number,
    "evaluation_metric": "fuzzy_match" | "bleu" | "gleu" | "meteor" | "cosine" | "rouge_1" | "rouge_2" | "rouge_3" | "rouge_4" | "rouge_5" | "rouge_l" 
}

• fuzzy_match: Fuzzy string match between input and reference, using rapidfuzz
• bleu: Computes the BLEU score between input and reference
• gleu: Computes the Google BLEU score between input and reference
• meteor: Computes the METEOR score between input and reference
• cosine: Computes Cosine similarity between embedded input and reference, using text-embedding-3-large. Only available for evals.
• rouge-*: Computes the ROUGE score between input and reference