科研 Skill

Hugging Face 社区评测

huggingface-community-evals

本地运行模型评测流程

官方收录发布者 huggingface-skills安全审核通过v0.1.0更新于 2026-06-08

Skill 说明

在本地 GPU 上用 inspect-ai / lighteval 跑 HF Hub 模型评测(支持 vLLM / Transformers / accelerate) | 输入:HF Hub 模型 ID + benchmark | 输出:评测分数 + 报告(inspect-ai / lighteval)

SKILL.md

Overview

This skill is for running evaluations against models on the Hugging Face Hub on local hardware.

It covers:

  • inspect-ai with local inference
  • lighteval with local inference
  • choosing between vllm, Hugging Face Transformers, and accelerate
  • smoke tests, task selection, and backend fallback strategy

It does not cover:

  • Hugging Face Jobs orchestration
  • model-card or model-index edits
  • README table extraction
  • Artificial Analysis imports
  • .eval_results generation or publishing
  • PR creation or community-evals automation

If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill.

If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to ~/code/community-evals.

All paths below are relative to the directory containing this SKILL.md.

When To Use Which Script

Use caseScript
Local inspect-ai eval on a Hub model via inference providersscripts/inspect_eval_uv.py
Local GPU eval with inspect-ai using vllm or Transformersscripts/inspect_vllm_uv.py
Local GPU eval with lighteval using vllm or acceleratescripts/lighteval_vllm_uv.py
Extra command patternsexamples/USAGE_EXAMPLES.md

Prerequisites

  • Prefer uv run for local execution.
  • Set HF_TOKEN for gated/private models.
  • For local GPU runs, verify GPU access before starting:
uv --version
printenv HF_TOKEN >/dev/null
nvidia-smi

If nvidia-smi is unavailable, either:

  • use scripts/inspect_eval_uv.py for lighter provider-backed evaluation, or
  • hand off to the hugging-face-jobs skill if the user wants remote compute.

Core Workflow

  1. Choose the evaluation framework.
    • Use inspect-ai when you want explicit task control and inspect-native flows.
    • Use lighteval when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.
  2. Choose the inference backend.
    • Prefer vllm for throughput on supported architectures.
    • Use Hugging Face Transformers (--backend hf) or accelerate as compatibility fallbacks.
  3. Start with a smoke test.
    • inspect-ai: add --limit 10 or similar.
    • lighteval: add --max-samples 10.
  4. Scale up only after the smoke test passes.
  5. If the user wants remote execution, hand off to hugging-face-jobs with the same script + args.

Quick Start

Option A: inspect-ai with local inference providers path

Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead.

uv run scripts/inspect_eval_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --task mmlu \
  --limit 20

Use this path when:

  • you want a quick local smoke test
  • you do not need direct GPU control
  • the task already exists in inspect-evals

Option B: inspect-ai on Local GPU

Best when you need to load the Hub model directly, use vllm, or fall back to Transformers for unsupported architectures.

Local GPU:

uv run scripts/inspect_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --task gsm8k \
  --limit 20

Transformers fallback:

uv run scripts/inspect_vllm_uv.py \
  --model microsoft/phi-2 \
  --task mmlu \
  --backend hf \
  --trust-remote-code \
  --limit 20

Option C: lighteval on Local GPU

Best when the task is naturally expressed as a lighteval task string, especially Open LLM Leaderboard style benchmarks.

Local GPU:

uv run scripts/lighteval_vllm_uv.py \
  --model meta-llama/Llama-3.2-3B-Instruct \
  --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
  --max-samples 20 \
  --use-chat-template

accelerate fallback:

uv run scripts/lighteval_vllm_uv.py \
  --model microsoft/phi-2 \
  --tasks "leaderboard|mmlu|5" \
  --backend accelerate \
  --trust-remote-code \
  --max-samples 20

Remote Execution Boundary

This skill intentionally stops at local execution and backend selection.

If the user wants to:

  • run these scripts on Hugging Face Jobs
  • pick remote hardware
  • pass secrets to remote jobs
  • schedule recurring runs
  • inspect / cancel / monitor jobs

then switch to the hugging-face-jobs skill and pass it one of these scripts plus the chosen arguments.

Task Selection

inspect-ai examples:

  • mmlu
  • gsm8k
  • hellaswag
  • arc_challenge
  • truthfulqa
  • winogrande
  • humaneval

lighteval task strings use suite|task|num_fewshot:

  • leaderboard|mmlu|5
  • leaderboard|gsm8k|5
  • leaderboard|arc_challenge|25
  • lighteval|hellaswag|0

Multiple lighteval tasks can be comma-separated in --tasks.

Backend Selection

  • Prefer inspect_vllm_uv.py --backend vllm for fast GPU inference on supported architectures.
  • Use inspect_vllm_uv.py --backend hf when vllm does not support the model.
  • Prefer lighteval_vllm_uv.py --backend vllm for throughput on supported models.
  • Use lighteval_vllm_uv.py --backend accelerate as the compatibility fallback.
  • Use inspect_eval_uv.py when Inference Providers already cover the model and you do not need direct GPU control.

Hardware Guidance

Model sizeSuggested local hardware
< 3Bconsumer GPU / Apple Silicon / small dev GPU
3B - 13Bstronger local GPU
13B+high-memory local GPU or hand off to hugging-face-jobs

For smoke tests, prefer cheaper local runs plus --limit or --max-samples.

Troubleshooting

  • CUDA or vLLM OOM:
    • reduce --batch-size
    • reduce --gpu-memory-utilization
    • switch to a smaller model for the smoke test
    • if necessary, hand off to hugging-face-jobs
  • Model unsupported by vllm:
    • switch to --backend hf for inspect-ai
    • switch to --backend accelerate for lighteval
  • Gated/private repo access fails:
    • verify HF_TOKEN
  • Custom model code required:
    • add --trust-remote-code

Examples

See:

  • examples/USAGE_EXAMPLES.md for local command patterns
  • scripts/inspect_eval_uv.py
  • scripts/inspect_vllm_uv.py
  • scripts/lighteval_vllm_uv.py

常见问题

Hugging Face 社区评测能做什么?

在本地 GPU 上用 inspect-ai / lighteval 跑 HF Hub 模型评测(支持 vLLM / Transformers / accelerate) | 输入:HF Hub 模型 ID + benchmark | 输出:评测分数 + 报告(inspect-ai / lighteval)。

如何安装Hugging Face 社区评测?

在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「Hugging Face 社区评测」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。

Hugging Face 社区评测需要哪些权限?

该科研 Skill声明需要:本地文件。安装前会再次列出,由你确认后才会生效。

Hugging Face 社区评测经过官方审核吗?

是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。