HomeAbout

Grader

Grader

String Grader

Simple pass or fail grader that returns 0 or 1.

  • eq = Returns 1 if the input matches the reference (case-sensitive), 0 otherwise
  • neq = Returns 1 if the input does not match the reference (case-sensitive), 0 otherwise
  • like = Returns 1 if the input contains the reference (case-sensitive), 0 otherwise
  • ilike = Returns 1 if the input contains the reference (not case-sensitive), 0 otherwise
{
    "type": "string_check",
    "name": string,
    "operation": "eq" | "ne" | "like" | "ilike",
    "input": string,
    "reference": string,
}

Text Similarity Grader

{
    "type": "text_similarity",
    "name": string,
    "input": string,
    "reference": string,
    "pass_threshold": number,
    "evaluation_metric": "fuzzy_match" | "bleu" | "gleu" | "meteor" | "cosine" | "rouge_1" | "rouge_2" | "rouge_3" | "rouge_4" | "rouge_5" | "rouge_l" 
}
  • fuzzy_match: Fuzzy string match between input and reference, using rapidfuzz
  • bleu: Computes the BLEU score between input and reference
  • gleu: Computes the Google BLEU score between input and reference
  • meteor: Computes the METEOR score between input and reference
  • cosine: Computes Cosine similarity between embedded input and reference, using text-embedding-3-large. Only available for evals.
  • rouge-*: Computes the ROUGE score between input and reference
AboutContact