LLMOps

尽管对于任何希望交付强大的 LLM 应用程序的人来说，评估大型语言模型 (LLM) 的输出都是必不可少的，但 LLM 评估对许多人来说仍然是一项艰巨的任务。无论您是通过微调来提高模型的准确性，还是增强检索增强生成 (RAG) 系统的上下文相关性，了解如何为您的用例开发和决定适当的 LLM 评估指标集对于构建坚不可摧的 LLM 评估管道都是必不可少的。

本文将教您有关 LLM 评估指标的所有知识，并附上代码示例。我们将深入探讨：

• 什么是 LLM 评估指标，如何使用它们来评估 LLM 系统，常见的陷阱，以及是什么让优秀的 LLM 评估指标变得优秀。
• 对 LLM 评估指标进行评分的所有不同方法，以及为什么 LLM-as-a-judge 最适合 LLM 评估。
• 如何使用 DeepEval (⭐https://github.com/confident-ai/deepeval) 实现并决定在代码中使用的适当的 LLM 评估指标集。
  您准备好阅读这份长长的清单了吗？让我们开始吧。

（更新：如果您正在寻找评估 LLM 聊天机器人/对话的指标，请查看这篇新文章！）

评估方法衡量我们的系统性能如何。对每个摘要进行人工评估（人工审查）既费时又昂贵，而且不可扩展，因此通常会辅以自动评估。许多自动评估方法试图衡量人类评估者会考虑的文本质量。这些品质包括流畅性、连贯性、相关性、事实一致性和公平性。内容或风格与参考文本的相似性也可能是生成文本的重要质量。

下图包括用于评估LLM生成内容的许多指标，以及如何对其进行分类。

https://github.com/NVIDIA/NeMo-Guardrails

NeMo Guardrails是一个开源工具包，用于轻松地将可编程护栏添加到基于LLM的会话系统中。

https://github.com/guardrails-ai/guardrails

为大型语言模型添加护栏。

https://github.com/truera/trulens

Evaluation and Tracking for LLM Experiments

Large Language Models (LLMs) are a type of artificial intelligence model that can generate human-like text. They are trained on large amounts of text data and can be used for a variety of natural language processing tasks, such as language translation, question answering, and text generation.

Evaluating LLMs is important to ensure that they are performing well and generating high-quality text. This is especially important for applications where the generated text is used to make decisions or provide information to users.

Over the past year, excitement around Large Language Models (LLMs) skyrocketed. With ChatGPT and BingChat, we saw LLMs approach human-level performance in everything from performance on standardized exams to generative art. However, many of these LLM-based features are new and have a lot of unknowns, hence require careful release to preserve privacy and social responsibility. While offline evaluation is suitable for early development of features, it cannot assess how model changes benefit or degrade the user experience in production.

The question answering system, that is based on semantic search and LLM currently one of the most popular application of LLM functionality. But what after we build it? How to evaluate the work of QnA system?

Although evaluating the outputs of Large Language Models (LLMs) is essential for anyone looking to ship robust LLM applications, LLM evaluation remains a challenging task for many. Whether you are refining a model’s accuracy through fine-tuning or enhancing a Retrieval-Augmented Generation (RAG) system’s contextual relevancy, understanding how to develop and decide on the appropriate set of LLM evaluation metrics for your use case is imperative to building a bulletproof LLM evaluation pipeline.

• Prerequisites
• Answering questions is complicated