Install promptfoo with Homebrew, Nix, winget

Project links

• Homepage
• Package manager page

Commands and aliases

• pf
• promptfoo

Project history

Promptfoo began as developer tooling for systematically comparing prompts and model outputs. Its scope expanded from prompt evaluation into testing agents and RAG pipelines, automated red teaming, vulnerability scanning, model auditing, and code-scanning workflows.

Adoption history

Promptfoo states that the underlying tooling was originally built for LLM applications serving more than ten million users in production. It subsequently developed into an open-source CLI and library usable locally, in application code, and in CI/CD.

The official repository now documents installation through npm, npx, pip, and Homebrew. In 2026, the project announced that Promptfoo had become part of OpenAI while remaining open source and MIT licensed.

How it is used

Users define prompts, providers, test variables, assertions, and red-team targets in configuration, then run evaluations from the CLI and inspect matrix or vulnerability reports.

API credentials are normally supplied through process environment variables or an environment file. Official guidance warns against resolving secrets into exported evaluation configuration.

Promptfoo may be run as a globally installed CLI, through `npx`, as a library, or in CI/CD.

Why package nerds care

Promptfoo brought familiar test-driven development practices—declarative cases, assertions, caching, matrices, and CI checks—to prompt and model evaluation. It is notable for combining quality evaluation and adversarial red teaming in a locally runnable package rather than requiring a hosted notebook or evaluation service.

Timeline

• Early project: Tooling originated in production LLM applications serving more than ten million users.
• Open-source expansion: Added declarative evaluations, multiple model providers, CI/CD use, and local reports.
• Later development: Expanded into red teaming, vulnerability scanning, RAG and agent evaluation, and model security.
• 2026: Promptfoo became part of OpenAI while remaining open source under the MIT license.

Related projects

• Promptfoo integrates with providers including OpenAI, Anthropic, Azure, Google, Hugging Face, and locally or privately implemented APIs.
• Its workflow complements general test runners and CI systems by treating prompts, providers, test variables, and assertions as declarative evaluation inputs.

Sources

• Official CLI documentation: https://www.promptfoo.dev/docs/usage/command-line/
• Official configuration documentation: https://www.promptfoo.dev/docs/configuration/guide/
• Official introduction: https://www.promptfoo.dev/docs/intro/
• Official repository README: https://github.com/promptfoo/promptfoo