promptfoo を Homebrew, Nix, winget でインストール

プロジェクトリンク

• ホームページ
• パッケージマネージャページ

コマンドとエイリアス

• pf
• promptfoo

プロジェクトの歴史

Promptfoo began as developer tooling for systematically comparing prompts and model outputs. Its scope expanded from prompt evaluation into testing agents and RAG pipelines, automated red teaming, vulnerability scanning, model auditing, and code-scanning workflows.

採用の歴史

Promptfoo states that the underlying tooling was originally built for LLM applications serving more than ten million users in production. It subsequently developed into an open-source CLI and library usable locally, in application code, and in CI/CD.

The official repository now documents installation through npm, npx, pip, and Homebrew. In 2026, the project announced that Promptfoo had become part of OpenAI while remaining open source and MIT licensed.

使われ方

Users define prompts, providers, test variables, assertions, and red-team targets in configuration, then run evaluations from the CLI and inspect matrix or vulnerability reports.

API credentials are normally supplied through process environment variables or an environment file. Official guidance warns against resolving secrets into exported evaluation configuration.

Promptfoo may be run as a globally installed CLI, through `npx`, as a library, or in CI/CD.

パッケージ好きにとっての重要性

Promptfoo brought familiar test-driven development practices—declarative cases, assertions, caching, matrices, and CI checks—to prompt and model evaluation. It is notable for combining quality evaluation and adversarial red teaming in a locally runnable package rather than requiring a hosted notebook or evaluation service.

タイムライン

• Early project: Tooling originated in production LLM applications serving more than ten million users.
• Open-source expansion: Added declarative evaluations, multiple model providers, CI/CD use, and local reports.
• Later development: Expanded into red teaming, vulnerability scanning, RAG and agent evaluation, and model security.
• 2026: Promptfoo became part of OpenAI while remaining open source under the MIT license.

Related projects

• Promptfoo integrates with providers including OpenAI, Anthropic, Azure, Google, Hugging Face, and locally or privately implemented APIs.
• Its workflow complements general test runners and CI systems by treating prompts, providers, test variables, and assertions as declarative evaluation inputs.

ソース

• Official CLI documentation: https://www.promptfoo.dev/docs/usage/command-line/
• Official configuration documentation: https://www.promptfoo.dev/docs/configuration/guide/
• Official introduction: https://www.promptfoo.dev/docs/intro/
• Official repository README: https://github.com/promptfoo/promptfoo