From Perplexity to ROUGE: Essential Metrics for Evaluating LLMs (Easy to Understand!)

#11 Advanced Generative AI: Understanding essential evaluation metrics.

Introduction

Large Language Models (LLMs) are becoming increasingly powerful and influential, permeating various domains. This expansion underscores the need for strong evaluation techniques to confirm their dependability and efficacy. By applying metrics, we can evaluate each LLM's performance and compare various models comprehensively, guaranteeing that we employ the most efficient language technology available. In this blog, we will explore popular LLM evaluation metrics in detail.

Photo by William Warby on Unsplash

Why are LLM metrics important?

As large language models (LLMs) continue to expand in scale, parameter count, and task versatility, concerns about their unpredictability and opaque nature intensify. LLMs' outputs are inherently non-deterministic, meaning semantically identical sentences can be expressed in varied ways. Given this, it becomes crucial to employ metrics as tools to assess the model's reliability and effectiveness. These functions provide a standardized means to evaluate LLMs' performance, ensuring their outputs are dependable and valuable for their intended applications.

What are Metrics?

Metrics are quantifiable indicators used to evaluate the performance and abilities of these models across different NLP tasks. They are instrumental in determining a model's accuracy, comparing various LLMs, and establishing benchmarks. These benchmarks set performance standards for new models to surpass, thereby fostering innovation and progress in the development of LLMs.

Exploring popular metrics available

1. Accuracy

In the context of Large Language Models (LLMs), accuracy is a crucial metric used to evaluate the model's performance. Here are some key points about accuracy as a metric in the context of LLMs:

• Definition: Accuracy measures how well an LLM performs a task compared to a human-annotated or predefined standard. It is essentially a score that reflects the model's correctness in its predictions.

• Calculation: It is calculated as the ratio of correct predictions to the total number of predictions made, often expressed as a percentage. For example, in text classification tasks, accuracy would indicate how accurately the model can categorize text into the correct classes.

\(\text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}} \)

• Scale: Accuracy is typically measured on a scale from 0 to 1, where 1 indicates perfect performance (100% accuracy), and 0 indicates complete failure (0% accuracy).

• Primary Tasks: Accuracy is particularly relevant in tasks like classification (e.g., sentiment analysis) and named entity recognition, where the goal is to identify categories or entities in the text correctly.

2. Perplexity

Perplexity is a measure used to evaluate the performance of probabilistic models, such as language models, in natural language processing (NLP). It quantifies how well a model predicts a sample of text. In the context of Large Language Models (LLMs), perplexity is commonly used to assess the model's ability to understand and generate human-like language.

• Definition: Perplexity measures the uncertainty of a language model in predicting the next word or token in a sequence. It reflects how well the model has captured the patterns in the training data and its ability to generate or comprehend text.

• Calculation: The perplexity of an LLM is calculated as the exponential of the average negative log-likelihood of a sequence of words. A higher perplexity score indicates that the model is less certain about its predictions, while a lower score suggests greater confidence and a better understanding of the language structure.

\(\begin{align*} \text{Perplexity} &= 2^{-\frac{1}{N} \sum_{i=1}^{N} \log_2 p(w_i)} \\ \text{where} & \\ N & = \text{Total number of words in the text} \\ w_i & = \text{The } i\text{-th word in the text} \\ p(w_i) & = \text{Probability assigned by the model to the } i\text{-th word} \end{align*} \)

• Scale: Perplexity is typically expressed as a positive number. Lower values indicate better model performance, with a perplexity of 1 being the ideal score, representing perfect prediction accuracy. Higher values indicate greater uncertainty in the model's predictions.

• Primary Tasks: Perplexity is especially relevant in tasks such as casual language modelling and text generation, where the goal is for the model to accurately predict or generate sequences of text that resemble human language.

3. Bilingual Evaluation Understudy (BLEU)

The BLEU (Bilingual Evaluation Understudy) score is a widely used metric for evaluating the quality of machine-translated text against one or more reference translations. It measures how many words and phrases or n-grams (sequences of n words) match in the translated text compared to the reference translations, considering different lengths of n-grams to assess both the accuracy of individual words and the correctness of word sequences. Additionally, the BLEU score incorporates a brevity penalty to ensure that translations are accurate and of appropriate length, making it a valuable tool for assessing the effectiveness of language translation models in natural language processing (NLP).

• Definition: The BLEU score is a metric that quantifies the similarity between machine-generated translations and one or more reference translations. It evaluates the precision of n-grams in the translated text and applies a brevity penalty to discourage overly short translations.

• Calculation: The BLEU score is calculated by comparing the n-grams of the translated text with those of the reference translations, computing precision for different n-gram lengths, and then combining these precision scores using a geometric mean. A brevity penalty is applied if the translated text is shorter than the reference translations.

\(\begin{align*} \text{BLEU} &= \text{BP} \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right) \\ \text{where} & \\ \text{BP} & = \text{Brevity Penalty, penalizes overly short translations} \\ N & = \text{Maximum length of n-grams considered (usually 4)} \\ w_n & = \text{Weight for the } n\text{-th n-gram (often uniform, i.e., } 1/N\text{)} \\ p_n & = \text{Precision of the } n\text{-th n-gram,} \\ & \quad \text{calculated as the ratio of the number of n-grams in the} \\ & \quad \text{candidate translation that match the reference translation} \\ & \quad \text{to the total number of n-grams in the candidate translation} \end{align*} \)

• Scale: The BLEU score ranges from 0 to 1, where 1 indicates a perfect match between the translated text and the reference translations. Higher scores represent better translation quality, with 0 indicating no overlap.

• Primary Tasks: The BLEU score is used primarily in language translation and summarization. In language translation, it evaluates the quality of translations from one language to another. In text summarization, it assesses how well the summarized text captures the essential information of the original text.

4. Recall-Oriented Understudy for Gisting Evaluation (ROUGE)

The ROUGE score is a tool used to check how good a summary or translation is in the field of natural language processing (NLP). Unlike another tool called BLEU, which mainly looks at how precise a translation is, ROUGE looks at both how much important information is captured (recall) and how accurate the information is (precision) when comparing a machine-generated text to one or more texts written by humans. This makes ROUGE especially useful for tasks like creating summaries or translating texts, where it's important to get both the main ideas and the details right.

• Definition: ROUGE measures the overlap between the generated text and the reference text(s), considering both the content and the structure. It aims to capture how well the generated text aligns with the reference text(s) in terms of accuracy and completeness.

• Main Variants:

  • ROUGE-N: Evaluates the overlap of n-grams between the generated and reference texts, calculating both recall (the proportion of n-grams in the reference that appear in the generated text) and precision (the proportion of n-grams in the generated text that appear in the reference).

  • ROUGE-L: It uses the Longest Common Subsequence (LCS) – the longest string of words appearing in the same order in both texts – to measure similarity, emphasizing the overall structure and flow of the content.

  • ROUGE-W: Similar to ROUGE-L, it considers LCS but assigns more weight to longer subsequences, emphasizing the importance of capturing larger chunks of information in the correct order.

  • ROUGE-S: This variant goes beyond strict word order. It considers "skip-bigrams" – pairs of words that might have other words in between them in the sentence. This helps capture the meaning even if the phrasing differs slightly between the generated and reference texts.

• Scale: A higher ROUGE score signifies better agreement between the generated text and the reference text(s), indicating higher quality in terms of both accuracy and completeness.

• Tasks: The ROUGE score is primarily used in tasks such as automatic text summarization, where it evaluates how well the summarized text captures the essence of the original text, and in machine translation, where it assesses the quality of the translated text.

Conclusion

In conclusion, using various metrics to evaluate Large Language Models (LLMs) is crucial for assessing their performance and guiding their development. Metrics like accuracy, perplexity, BLEU, ROUGE, and BERTScore help us understand different aspects of LLMs, from their precision to their ability to generate coherent text. By leveraging these metrics, we can identify areas for improvement and drive innovation in the field. As LLMs continue to evolve, so will the ways we evaluate them, ensuring they remain effective tools for various tasks.

---