科研 Skill
Hugging Face 社区评测
huggingface-community-evals
本地运行模型评测流程
Skill 说明
在本地 GPU 上用 inspect-ai / lighteval 跑 HF Hub 模型评测(支持 vLLM / Transformers / accelerate) | 输入:HF Hub 模型 ID + benchmark | 输出:评测分数 + 报告(inspect-ai / lighteval)
SKILL.md
Overview
This skill is for running evaluations against models on the Hugging Face Hub on local hardware.
It covers:
inspect-aiwith local inferencelightevalwith local inference- choosing between
vllm, Hugging Face Transformers, andaccelerate - smoke tests, task selection, and backend fallback strategy
It does not cover:
- Hugging Face Jobs orchestration
- model-card or
model-indexedits - README table extraction
- Artificial Analysis imports
.eval_resultsgeneration or publishing- PR creation or community-evals automation
If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill.
If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to ~/code/community-evals.
All paths below are relative to the directory containing this
SKILL.md.
When To Use Which Script
| Use case | Script |
|---|---|
Local inspect-ai eval on a Hub model via inference providers | scripts/inspect_eval_uv.py |
Local GPU eval with inspect-ai using vllm or Transformers | scripts/inspect_vllm_uv.py |
Local GPU eval with lighteval using vllm or accelerate | scripts/lighteval_vllm_uv.py |
| Extra command patterns | examples/USAGE_EXAMPLES.md |
Prerequisites
- Prefer
uv runfor local execution. - Set
HF_TOKENfor gated/private models. - For local GPU runs, verify GPU access before starting:
uv --version
printenv HF_TOKEN >/dev/null
nvidia-smi
If nvidia-smi is unavailable, either:
- use
scripts/inspect_eval_uv.pyfor lighter provider-backed evaluation, or - hand off to the
hugging-face-jobsskill if the user wants remote compute.
Core Workflow
- Choose the evaluation framework.
- Use
inspect-aiwhen you want explicit task control and inspect-native flows. - Use
lightevalwhen the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.
- Use
- Choose the inference backend.
- Prefer
vllmfor throughput on supported architectures. - Use Hugging Face Transformers (
--backend hf) oraccelerateas compatibility fallbacks.
- Prefer
- Start with a smoke test.
inspect-ai: add--limit 10or similar.lighteval: add--max-samples 10.
- Scale up only after the smoke test passes.
- If the user wants remote execution, hand off to
hugging-face-jobswith the same script + args.
Quick Start
Option A: inspect-ai with local inference providers path
Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead.
uv run scripts/inspect_eval_uv.py \
--model meta-llama/Llama-3.2-1B \
--task mmlu \
--limit 20
Use this path when:
- you want a quick local smoke test
- you do not need direct GPU control
- the task already exists in
inspect-evals
Option B: inspect-ai on Local GPU
Best when you need to load the Hub model directly, use vllm, or fall back to Transformers for unsupported architectures.
Local GPU:
uv run scripts/inspect_vllm_uv.py \
--model meta-llama/Llama-3.2-1B \
--task gsm8k \
--limit 20
Transformers fallback:
uv run scripts/inspect_vllm_uv.py \
--model microsoft/phi-2 \
--task mmlu \
--backend hf \
--trust-remote-code \
--limit 20
Option C: lighteval on Local GPU
Best when the task is naturally expressed as a lighteval task string, especially Open LLM Leaderboard style benchmarks.
Local GPU:
uv run scripts/lighteval_vllm_uv.py \
--model meta-llama/Llama-3.2-3B-Instruct \
--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
--max-samples 20 \
--use-chat-template
accelerate fallback:
uv run scripts/lighteval_vllm_uv.py \
--model microsoft/phi-2 \
--tasks "leaderboard|mmlu|5" \
--backend accelerate \
--trust-remote-code \
--max-samples 20
Remote Execution Boundary
This skill intentionally stops at local execution and backend selection.
If the user wants to:
- run these scripts on Hugging Face Jobs
- pick remote hardware
- pass secrets to remote jobs
- schedule recurring runs
- inspect / cancel / monitor jobs
then switch to the hugging-face-jobs skill and pass it one of these scripts plus the chosen arguments.
Task Selection
inspect-ai examples:
mmlugsm8khellaswagarc_challengetruthfulqawinograndehumaneval
lighteval task strings use suite|task|num_fewshot:
leaderboard|mmlu|5leaderboard|gsm8k|5leaderboard|arc_challenge|25lighteval|hellaswag|0
Multiple lighteval tasks can be comma-separated in --tasks.
Backend Selection
- Prefer
inspect_vllm_uv.py --backend vllmfor fast GPU inference on supported architectures. - Use
inspect_vllm_uv.py --backend hfwhenvllmdoes not support the model. - Prefer
lighteval_vllm_uv.py --backend vllmfor throughput on supported models. - Use
lighteval_vllm_uv.py --backend accelerateas the compatibility fallback. - Use
inspect_eval_uv.pywhen Inference Providers already cover the model and you do not need direct GPU control.
Hardware Guidance
| Model size | Suggested local hardware |
|---|---|
< 3B | consumer GPU / Apple Silicon / small dev GPU |
3B - 13B | stronger local GPU |
13B+ | high-memory local GPU or hand off to hugging-face-jobs |
For smoke tests, prefer cheaper local runs plus --limit or --max-samples.
Troubleshooting
- CUDA or vLLM OOM:
- reduce
--batch-size - reduce
--gpu-memory-utilization - switch to a smaller model for the smoke test
- if necessary, hand off to
hugging-face-jobs
- reduce
- Model unsupported by
vllm:- switch to
--backend hfforinspect-ai - switch to
--backend accelerateforlighteval
- switch to
- Gated/private repo access fails:
- verify
HF_TOKEN
- verify
- Custom model code required:
- add
--trust-remote-code
- add
Examples
See:
examples/USAGE_EXAMPLES.mdfor local command patternsscripts/inspect_eval_uv.pyscripts/inspect_vllm_uv.pyscripts/lighteval_vllm_uv.py
相关科研 Skill
中药材分子鉴定用ITS2等DNA条形码鉴定药材基原物种,处理多基原与降解样品局限。适用于药材真伪鉴别研究,不替代法定检验。
中药网络药理学用TCMSP等数据库构建成分-靶点-通路网络并做富集分析,批判性讨论方法学局限。适用于中药机制假设生成与论文方法复核。
中医术语标准与知识图谱建立中医药术语与WHO/ICD-11标准的映射,设计本体并用Neo4j构建知识图谱。适用于数据整合与科研检索,不作临床决策系统。
方剂配伍规律挖掘对方剂做药名规范化、关联规则与网络聚类分析,发现候选核心药组。适用于用药规律挖掘,结果仅为待验证假设,不作处方依据。
证候与症状数据分析对症状量表数据做术语规范、潜在类别与因子分析,探索统计学证候模式。适用于证候分型研究,结果不用于个体辨证诊断。
泵用混合物热力学计算Calculates thermodynamic properties and mixture phase behavior for pump applications using the thermo Python library’s DIPPR/NIST correlations. Use for VLE or mixture property calculations.
常见问题
Hugging Face 社区评测能做什么?
在本地 GPU 上用 inspect-ai / lighteval 跑 HF Hub 模型评测(支持 vLLM / Transformers / accelerate) | 输入:HF Hub 模型 ID + benchmark | 输出:评测分数 + 报告(inspect-ai / lighteval)。
如何安装Hugging Face 社区评测?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「Hugging Face 社区评测」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
Hugging Face 社区评测需要哪些权限?
该科研 Skill声明需要:本地文件。安装前会再次列出,由你确认后才会生效。
Hugging Face 社区评测经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。