科研 Skill
多模态大模型临床评测设计
mllm-eval
Audits how an LLM or multimodal LLM task is evaluated, covering reference standard and hallucination. Use when reviewing an evaluation harness for a clinical AI model.
SKILL.md
MLLM-Eval Skill
Purpose
This skill makes an LLM / MLLM clinical evaluation defensible: a real adjudicated reference standard, faithfulness measured not assumed, clinical-efficacy metrics beyond n-gram overlap, a pretraining- contamination check, prompt-sensitivity disclosed, and a reader study where text is generated. It is model-agnostic — every check applies to a closed API and to open weights — and read-only (an advisory design/audit skill): it audits the evaluation design and specifies and routes the clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 via their published extractors) rather than running the model or computing the metrics itself.
It is the LLM/MLLM evaluation-design counterpart in the lane — an auditor that hands the specified
metrics to their extractors and /analyze-stats, parallel to how /model-assessment audits an imaging
model's design and computes its metrics. The reviewer-side
audit of a finished manuscript uses the mllm_evaluation.md (ME0–ME8) probe via /self-review and
/peer-review; this skill is the author-side harness design. It routes the reader study to
/design-ai-benchmarking, the sizing to /calc-sample-size, and TRIPOD-LLM / MI-CLEAR-LLM compliance to
/check-reporting.
When to use
- You are designing or auditing an evaluation of an LLM/MLLM on a clinical task and want it to cover the axes a reviewer will check (reference standard, faithfulness, contamination, prompt sensitivity, reader study).
When NOT to use
- AI-vs-human-expert benchmark with a rated rubric →
/design-ai-benchmarking. - Imaging prediction/segmentation model →
/model-assessment. - Image-to-image generative model → the
image_synthesisprobe. - Training / serving the LLM → out of scope.
- Item-level TRIPOD-LLM / MI-CLEAR-LLM audit of a finished manuscript →
/check-reporting.
Workflow
Phase 1 — Pin the task, model, comparator, decoding (ME0)
State the task (report generation / VQA / extraction-classification), the exact model + version/date (closed API or open-weights id), the decoding settings (temperature, seed, max tokens), and what the outputs are scored against.
Phase 2 — Reference standard + metrics (ME1, ME2)
Require an adjudicated expert reference (not a single unverified report or a model-derived label). For report generation, report a clinical-efficacy metric — RadGraph-F1 (Jain et al., NeurIPS 2021) or CheXbert-F1 (Smit et al., 2020), or the composite RadCliQ (Yu et al., Patterns 2023) — alongside any BLEU/ROUGE, with CIs. For VQA/classification, state the answer-matching rule and report per-class sensitivity/specificity (or precision/recall/F1) and PPV at the real prevalence, with CIs; accuracy only alongside them — at 2% prevalence, answering "negative" every time scores 98% accuracy.
Phase 3 — Faithfulness + contamination (ME3, ME4)
Add an atomic-fact faithfulness measure + a false-premise / abstention probe (MedVH, Med-HALT) — report a hallucination rate, not just accuracy. For any public benchmark (VQA-RAD, SLAKE, MIMIC-CXR- derived, MedQA), add a contamination statement: training cutoff vs benchmark release, a held-out / post-cutoff set, or a contamination probe.
Phase 4 — Prompt sensitivity + reader study (ME5, ME7)
Disclose the exact prompt(s), temperature/seed, ≥ 3 runs with variance, and a prompt-robustness
check. For a deployment/utility claim, design a blinded reader study with an error taxonomy (route
the rubric/IRR to /design-ai-benchmarking, ICC/κ to /analyze-stats, sizing to /calc-sample-size).
Phase 5 — Gate the plan (deterministic)
Declare the axes in eval_manifest.json (copy templates/eval_manifest.json;
fields and allowed values in references/eval_manifest_schema.md), then:
python3 scripts/check_mllm_eval_completeness.py \
--manifest eval_manifest.json --strict
Use "none" for an axis not done and "other:<description>" for a method not listed; any other
value exits 2. --plan plan.md --task report_generation|vqa|classification still runs the older
keyword check on prose (see Known limits).
NGRAM_ONLY / FAITHFULNESS_MISSING / REFERENCE_STANDARD_MISSING / CONTAMINATION_UNADDRESSED /
READER_STUDY_MISSING must be resolved. A classification manifest always reports
CLASSIFICATION_METRICS_NOT_ASSESSED (Minor): the manifest has no metric field for it, so check
per-class sensitivity/specificity and PPV at the real prevalence by eye.
Phase 6 — Hand off
Methods/Results → /write-paper; compliance (TRIPOD-LLM / MI-CLEAR-LLM) → /check-reporting; reviewer
audit → /self-review (loads ME0–ME8).
Anti-Hallucination
- Never fabricate model outputs, reference labels, or metric scores. Compute only what the supplied outputs allow; if a reference standard or outputs are missing, say so and stop.
- Never report n-gram overlap (BLEU/ROUGE) as clinical correctness — pair it with a clinical-efficacy metric, and flag the n-gram score for what it is.
- Never claim "no contamination" without a stated check when a public benchmark is used.
- If a metric (RadGraph-F1 / CheXbert-F1) or its extractor is uncertain, flag
[VERIFY]and ask rather than inventing a number.
Deterministic gate
scripts/check_mllm_eval_completeness.py — task-aware presence gate on the evaluation plan (stdlib,
network-free). Reproducible challenge:
bash scripts/mllm_eval_completeness_challenge/verify.sh.
Known limits. Manifest mode checks what is declared, not that the work was done; keep the
manifest in step with the Methods. It also does not check that a declared metric suits the task:
report-generation metrics (BLEU, RadGraph F1) declared for a VQA task are accepted without
comment. Prose mode (--plan) checks that a term is present; it does not read negation or sense.
"No human evaluation was performed" or "hallucination was not assessed" still counts as covering
that axis, and a word used in another sense still clears it: "green" anywhere clears the clinical-metric check, "unsupported"
clears faithfulness, "data leakage" from a patient split clears contamination, and "ground truth"
clears the reference standard whatever it refers to. Read each cleared axis in the plan yourself
before treating an exit 0 as clean.
Boundaries
mllm-eval (this skill: harness design + completeness gate, model-agnostic)
├─ design-ai-benchmarking (reader-study rubric / IRR)
├─ calc-sample-size (reader + case sizing)
├─ write-paper + check-reporting (TRIPOD-LLM / MI-CLEAR-LLM)
└─ self-review / peer-review (ME0–ME8 reviewer probe)
Reference Files
references/evaluation_axes.md— the why behind the ME2–ME7 axes: clinical-efficacy metrics beyond n-gram overlap (e.g. RadGraph-F1 / CheXbert-F1 vs BLEU/ROUGE), faithfulness & hallucination, pretraining/benchmark contamination, prompt-sensitivity & determinism, answer-matching, and the reader study — each mapped to its gate verdict. Load on demand during Phases 2–4.
示例提示词
帮我设计一套评测多模态大模型生成放射报告质量的方案。
所属插件
安装该插件会一并获得同领域的 58 个 Skill / MCP。
同插件的其他科研 Skill
- 医学AI论文检索优化Optimizes a medical AI paper’s title, abstract and metadata for AI search engines. Use when a paper should be findable and citable by RAG tools and AI search.
- 临床数据统计分析与建模Generates reproducible Python or R code for clinical statistics and tables. Use when collected data needs Table 1, regression, survival or agreement analysis.
- 作者发表记录与策略分析Analyzes a PubMed author record to classify study types and authorship patterns. Use when planning a publication strategy from a researcher’s track record.
- 批量队列暴露结局分析Generates one cohort-analysis script per exposure-outcome pair from one template. Use when a validated method must run across many variable combinations at once.
- 样本量估算与检验效能Walks a decision tree to the right test and computes sample size with IRB text. Use when planning how many patients a study needs before data collection.
- 因果推断方法设计Runs DAG-based, instrumental-variable, difference-in-differences or RD analyses. Use when an association in observational data must support a causal claim.
- 报告规范逐条核查Audits a manuscript item by item against 49 reporting and risk-of-bias checklists. Use when checking a draft against STROBE, CONSORT, TRIPOD or similar guidelines.
- 临床数据清洗与画像Profiles a clinical dataset and generates cleaning code in staged, approved steps. Use when a CSV or Excel dataset needs missing-value and outlier checks before analysis.
- 跨国队列数据协调对比Harmonizes variables and runs parallel weighted analyses across national surveys like KNHANES and NHANES. Use when comparing one exposure-outcome link across countries.
- 变量定义与截断值溯源Maps exposure, outcome and covariate definitions to a cited guideline and DB columns. Use when variable cutoffs need a citable basis before the study protocol.
- 本地化临床数据脱敏Runs a local, network-free script that detects and removes identifiers from clinical data. Use when data may contain PHI and must be de-identified before LLM analysis.
- AI对标专家基准研究设计Plans the arms, rubrics, calibration probes and reviewer panel for an AI-vs-expert benchmark. Use before data collection when designing an AI-vs-human rating study.
- 放射影像AI研究设计审查Reviews a study’s analysis unit, cohort logic, leakage risk and reporting-guideline fit. Use when checking a clinical or medical-AI study design before drafting.
- 传染病仓室模型拟合Builds and fits SIR, SEIR and SEIHR compartmental models with Rt estimation. Use when an outbreak or epidemic’s spread needs mathematical modeling in Python.
- ICMJE利益冲突声明填写Clones a pre-filled synthetic ICMJE COI form, replacing only date, name and title. Use when each author on a submission needs a disclosure form filled quickly.
- 机构Word表单内容填充Fills an institutional Word form, such as an IRB protocol, without breaking its layout. Use when drafted content must be rendered into a CJK-aware .docx template.
- 队列数据库选题挖掘Profiles a longitudinal cohort database and ranks research-topic proposals with gap evidence. Use when looking for a question an available cohort or registry can answer.
- 投稿期刊匹配推荐Matches a manuscript abstract against curated journal profiles and ranks candidates. Use when choosing where to submit, or re-targeting after a rejection.
- DOI全文批量获取Batch-downloads open-access PDFs for a list of DOIs via Unpaywall, PMC and OpenAlex. Use when full texts are needed for a screening set such as a meta-analysis.
- 数据集编码本自动生成Profiles every variable in a tabular dataset into a structured, citable codebook. Use when a CSV, Excel or Stata dataset needs a data dictionary before analysis.
- 全球卫生数据获取分析Accesses and analyzes global health data from WHO GHO, IHME GBD and Our World in Data. Use when DALYs, age-standardized rates or choropleth maps are needed.
- 医学AI项目基金申请书撰写Structures a grant proposal’s significance, innovation, approach, aims and milestones. Use when drafting a research funding proposal, including a Korean grant plan.
- 卫生经济学评价建模Builds Markov cohort models with QALYs, PSA and cost-effectiveness planes. Use when a health intervention needs a formal cost-effectiveness evaluation.
- AI写作痕迹识别改写Scans a manuscript for AI writing patterns and rewrites flagged passages carefully. Use when a draft or response letter reads as obviously AI-written.
常见问题
多模态大模型临床评测设计能做什么?
Audits how an LLM or multimodal LLM task is evaluated, covering reference standard and hallucination. Use when reviewing an evaluation harness for a clinical AI model。
如何安装多模态大模型临床评测设计?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「多模态大模型临床评测设计」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
多模态大模型临床评测设计需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
多模态大模型临床评测设计经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。