使用 Homebrew, Nix, winget 安装 promptfoo

项目链接

• 主页
• 软件包管理器页面

命令和别名

• pf
• promptfoo

项目历史

Promptfoo began as developer tooling for systematically comparing prompts and model outputs. Its scope expanded from prompt evaluation into testing agents and RAG pipelines, automated red teaming, vulnerability scanning, model auditing, and code-scanning workflows.

采用历史

Promptfoo states that the underlying tooling was originally built for LLM applications serving more than ten million users in production. It subsequently developed into an open-source CLI and library usable locally, in application code, and in CI/CD.

The official repository now documents installation through npm, npx, pip, and Homebrew. In 2026, the project announced that Promptfoo had become part of OpenAI while remaining open source and MIT licensed.

使用方式

Users define prompts, providers, test variables, assertions, and red-team targets in configuration, then run evaluations from the CLI and inspect matrix or vulnerability reports.

API credentials are normally supplied through process environment variables or an environment file. Official guidance warns against resolving secrets into exported evaluation configuration.

Promptfoo may be run as a globally installed CLI, through `npx`, as a library, or in CI/CD.

为什么软件包爱好者会关心

Promptfoo brought familiar test-driven development practices—declarative cases, assertions, caching, matrices, and CI checks—to prompt and model evaluation. It is notable for combining quality evaluation and adversarial red teaming in a locally runnable package rather than requiring a hosted notebook or evaluation service.

时间线

• Early project: Tooling originated in production LLM applications serving more than ten million users.
• Open-source expansion: Added declarative evaluations, multiple model providers, CI/CD use, and local reports.
• Later development: Expanded into red teaming, vulnerability scanning, RAG and agent evaluation, and model security.
• 2026: Promptfoo became part of OpenAI while remaining open source under the MIT license.

Related projects

• Promptfoo integrates with providers including OpenAI, Anthropic, Azure, Google, Hugging Face, and locally or privately implemented APIs.
• Its workflow complements general test runners and CI systems by treating prompts, providers, test variables, and assertions as declarative evaluation inputs.

来源

• Official CLI documentation: https://www.promptfoo.dev/docs/usage/command-line/
• Official configuration documentation: https://www.promptfoo.dev/docs/configuration/guide/
• Official introduction: https://www.promptfoo.dev/docs/intro/
• Official repository README: https://github.com/promptfoo/promptfoo