科研 Skill
评分者间信度分析
inter-rater-reliability
Computes inter-rater/inter-coder reliability with Cohen’s/weighted/Fleiss’ kappa, Krippendorff’s alpha, ICC, and Gwet’s AC1. Use when raters coded the same cases and agreement must be reported.
SKILL.md
Inter-Rater Reliability
Purpose
Before using human-coded or human-rated data (behavioral observation, clinical diagnosis, content-analysis codes, essay/video ratings) in any downstream analysis, quantify how much raters agree. Percent agreement alone is misleading because it does not correct for chance agreement. This skill picks the right chance-corrected statistic for the data type (nominal, ordinal, binary, continuous; two raters or many; complete or missing data) and reports it with the interpretation thresholds the field actually uses.
When to Use
- Two or more coders independently rated/classified the same set of items, video clips, transcripts, or participants, and the write-up needs a reliability statistic before the "real" analysis.
- A clinical or diagnostic measure was scored by more than one rater (e.g., two clinicians scoring the same interview) and you need to report diagnostic agreement.
- A continuous rating (e.g., 1–7 Likert ratings of stimuli, essay scores) was produced by multiple judges and you need an ICC for reliability.
- Checking whether a coding scheme is reliable enough to hand off to a single coder for the rest of the dataset.
Does NOT trigger
- Scale/questionnaire internal-consistency reliability across items
(Cronbach's alpha, omega) — use
psychometricsoralterlab-sem-psychometrics. - Test-retest reliability of a single rater's scores over time — that is a different reliability question (stability, not agreement across raters); a simple correlation or ICC(3,1) design still applies but interpret it as temporal stability, not inter-rater agreement.
Choosing the Right Statistic
| Data type | # Raters | Same raters for all items? | Missing data? | Use |
|---|---|---|---|---|
| Binary / nominal | 2 | Yes | No | Cohen's kappa |
| Ordinal (e.g., severity 1–4) | 2 | Yes | No | Weighted kappa (linear or quadratic weights) |
| Nominal | ≥ 3 | Yes | No | Fleiss' kappa |
| Any (nominal/ordinal/interval/ratio) | ≥ 2 | Not necessarily | Yes, allowed | Krippendorff's alpha |
| Continuous / interval rating | ≥ 2 | Yes | No | Intraclass correlation (ICC) |
| Any, with known prevalence/bias concerns | 2 | Yes | No | Gwet's AC1 (robust to the "kappa paradox" when prevalence is extreme) |
The kappa paradox: Cohen's/Fleiss' kappa can be low even when raw agreement is very high, if the trait being rated is rare or very common (skewed prevalence) or if raters are both highly accurate but the base rate is imbalanced. When this happens, report Gwet's AC1 alongside kappa and say explicitly that the low kappa is a prevalence artifact, not poor rater agreement — do not silently pick whichever statistic looks better.
Prerequisites
- Python 3.10+ with
pandas,numpy,scipy,statsmodels;pip install statsmodels pandas numpy scipy. - For Krippendorff's alpha:
pip install krippendorff(pure-Python implementation that handles missing data and multiple metric types out of the box), or implement it by hand usingreferencesbelow if the package is unavailable. - For ICC:
pingouin(pip install pingouin) has a ready-madepingouin.intraclass_corrthat implements all six Shrout & Fleiss (1979) ICC forms — simpler and less error-prone than hand-rolling ANOVA sums of squares. - Know, before computing anything: how many raters, whether every rater rated every item (fully crossed) or only a subset (partially crossed/missing), and whether the rating scale is nominal, ordinal, or interval — this decides which row of the table above applies.
Workflow
Step 1 — Cohen's kappa and weighted kappa (2 raters, nominal/ordinal)
import numpy as np
import pandas as pd
from sklearn.metrics import cohen_kappa_score
# ratings: two columns, one row per item, categorical codes (strings or ints)
# df = pd.DataFrame({"rater_a": [...], "rater_b": [...]})
def cohens_kappa(df: pd.DataFrame, col_a: str, col_b: str, weights=None) -> dict:
"""
weights: None for unweighted (nominal) kappa, 'linear' or 'quadratic'
for ordinal categories (e.g., severity ratings on an ordered scale).
"""
kappa = cohen_kappa_score(df[col_a], df[col_b], weights=weights)
n = len(df)
po = (df[col_a] == df[col_b]).mean() # observed raw agreement
return {"kappa": round(kappa, 3), "raw_agreement": round(po, 3), "n": n}
result = cohens_kappa(df, "rater_a", "rater_b")
print(f"Cohen's kappa = {result['kappa']}, raw agreement = {result['raw_agreement']:.1%}")
Interpretation (Landis & Koch, 1977 — widely cited but treat as a rough guide, not a hard cutoff): < 0 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect.
Step 2 — Fleiss' kappa (3+ raters, nominal, every item rated by the same number of raters)
from statsmodels.stats.inter_rater import fleiss_kappa, aggregate_raters
# long_df: one row per (item, rater) with a 'category' column of codes
agg_table, categories = aggregate_raters(long_df[["item_id", "rater_id", "category"]])
kappa = fleiss_kappa(agg_table, method="fleiss")
print(f"Fleiss' kappa = {kappa:.3f} across {agg_table.shape[0]} items, "
f"{len(categories)} categories")
aggregate_raters expects raw (item × rater) long-format data and converts
it to the (item × category count) matrix fleiss_kappa needs — do not build
that count matrix by hand unless you have verified the category order.
Step 3 — Krippendorff's alpha (any data type, handles missing ratings)
import krippendorff
import numpy as np
# reliability_data: one row per rater, one column per item; np.nan for
# items a given rater did not rate. level_of_measurement must match the
# variable type or the result is meaningless.
alpha = krippendorff.alpha(
reliability_data=reliability_matrix, # shape (n_raters, n_items)
level_of_measurement="nominal", # or 'ordinal', 'interval', 'ratio'
)
print(f"Krippendorff's alpha = {alpha:.3f}")
Krippendorff's own guidance: alpha ≥ .800 reliable; .667–.800 tentative conclusions only; below .667 do not rely on the coding. These thresholds are stricter than the Landis & Koch kappa bands above — do not reuse one scale for the other statistic.
Step 4 — ICC for continuous ratings
import pingouin as pg
# long_df: one row per (item, rater) with a numeric 'rating' column
icc_table = pg.intraclass_corr(
data=long_df, targets="item_id", raters="rater_id", ratings="rating"
)
print(icc_table.set_index("Type"))
# Report ICC(2,k) for "average of these specific raters generalizes to
# other items rated by these same raters" (consistency across *items*),
# or ICC(2,1) for the reliability of a SINGLE rater's score; choose the
# form that matches how the ratings will actually be used downstream.
Picking the wrong ICC form is the most common misreport in this area: ICC(2,1) answers "how reliable is one rater's score", ICC(2,k) answers "how reliable is the mean of all k raters' scores" — these can differ substantially and are not interchangeable.
Step 5 — Gwet's AC1 (robust alternative when kappa looks paradoxically low)
# Gwet's AC1 for two raters, nominal data (hand computation — no mainstream
# Python package; port from Gwet, 2008, or compute in R with irrCAC::gwet.ac1.raw())
def gwet_ac1(df: pd.DataFrame, col_a: str, col_b: str, categories: list) -> float:
n = len(df)
po = (df[col_a] == df[col_b]).mean()
pi_bar = {
c: ((df[col_a] == c).sum() + (df[col_b] == c).sum()) / (2 * n)
for c in categories
}
q = len(categories)
pe = sum(p * (1 - p) for p in pi_bar.values()) / (q - 1) if q > 1 else 0.0
return (po - pe) / (1 - pe) if pe < 1 else float("nan")
ac1 = gwet_ac1(df, "rater_a", "rater_b", categories=sorted(df["rater_a"].unique()))
print(f"Gwet's AC1 = {ac1:.3f}")
If no vetted Python package is available in the environment for AC1, prefer
running irrCAC::gwet.ac1.raw() in R and reporting which implementation was
used, rather than trusting a from-scratch formula without cross-checking it
against a worked example from Gwet (2008) or Wongpakaran et al. (2013).
Output Template
STATISTIC: <Cohen's kappa / weighted kappa / Fleiss' kappa / Krippendorff's alpha / ICC(form) / Gwet's AC1>
VALUE: <point estimate> (raw agreement = <%>, n items = <n>, n raters = <k>)
INTERPRETATION: <qualitative band, with the specific guideline cited>
CAVEATS: <prevalence/kappa-paradox check; missing-data handling; which ICC form and why>
Limitations
- Chance-corrected statistics (kappa family) assume raters are independent; if raters discussed cases or one rater's codes were used to train the other, the statistic overstates true reliability.
- Kappa/Fleiss' kappa require every item to have the same, fixed set of possible categories known in advance; novel categories appearing in only one rater's codes will distort the marginal proportions used in the chance-agreement term.
- These statistics quantify agreement, not accuracy — high agreement between two biased raters does not mean the codes are correct against any external gold standard.
- Small numbers of items (rule of thumb: under ~20–30) give kappa/alpha wide confidence intervals; report a bootstrap or Krippendorff's own CI procedure rather than a bare point estimate when n is small.
Quality Checks
- Report raw percent agreement alongside the chance-corrected statistic so a reader can judge whether a low kappa reflects a prevalence artifact.
- State the exact ICC form (e.g., ICC(2,1) vs ICC(2,k)) and the one-way/two-way, random/fixed model assumptions used — "the ICC was .81" alone is not reproducible.
- When reliability is below the field's working threshold (e.g., kappa < .60, alpha < .667), recommend recoding a sample, refining the codebook, or retraining coders before proceeding to substantive analysis — do not silently continue with unreliable codes.
- Verify the category set used for chance-agreement calculations matches across all raters; a category one rater never used should still appear in the denominator if it was a codebook option.
References
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
- Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage.
- Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
- Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
示例提示词
我想算两位编码员对这批访谈记录打分的Cohen's kappa和一致性。
所属插件
安装该插件会一并获得同领域的 13 个 Skill / MCP。
同插件的其他科研 Skill
- 结构方程与测量模型拟合Fits CFA, SEM, EFA, and IRT models with semopy and factor_analyzer, reporting fit indices, omega reliability, and invariance. Use for latent-variable, factor-loading, or invariance requests.
- 临床量表评分解读Scores PHQ-9, GAD-7, PCL-5, BDI-II, and SWLS, computing severity, prorated scores, and the Reliable Change Index. Use when batch-scoring clinical scales or testing pre-post change.
- 实验设计与顺序平衡Guides within- vs between-subjects design, Latin-square and ABBA counterbalancing, confound control, and preregistration before data collection. Use when planning an experiment’s design.
- 眼动数据分析Processes raw eye-tracking data: IVT fixation detection, AOI dwell time, scanpath similarity, heatmaps, pupillometry baseline correction. Use for gaze or pupil data from eye trackers.
- 心理学元分析Pools effect sizes across studies with Cohen’s d, Hedges’ g, random-effects pooling, forest/funnel plots, and publication-bias tests. Use when synthesizing results from multiple studies.
- 多层线性混合效应模型Fits multilevel/mixed-effects models with pymer4 and lme4: ICC, random slopes, likelihood-ratio comparison, within-person centering for ESM data. Use for nested or repeated-measures designs.
- 心理学研究功效分析Computes power, sample size, or minimum detectable effect for t-tests, ANOVA, correlation, regression, and simulation-based designs including mediation. Use when planning a study’s sample size.
- 心理测量学分析Runs CTT item stats, Cronbach’s alpha, EFA, CFA, 2PL IRT, measurement invariance, and Mantel-Haenszel DIF detection in Python. Use when validating a scale’s reliability, structure, or item fairness.
- PsychoPy行为实验设计Builds PsychoPy experiments: monitor calibration, stimulus presentation, response timing, BIDS events.tsv output, Pavlovia deployment checklist. Use when implementing a stimulus-response task.
- 反应时数据分析Cleans RT data and fits ex-Gaussian distributions, EZ-diffusion parameters, and Bayesian drift-diffusion models with PyMC. Use for RT outliers, distribution shape, or decision-model parameters.
- 单被试实验设计分析Analyzes single-case ABAB, multiple-baseline, and alternating-treatment designs with visual analysis, Tau-U, PND, and randomization tests. Use for studies with few participants observed across phases.
- 问卷量表翻译审查Reviews OB, management, and psychology survey scales for translation equivalence and adaptation validity, producing a Chinese dashboard report. Use for pre-launch questionnaire quality checks.
常见问题
评分者间信度分析能做什么?
Computes inter-rater/inter-coder reliability with Cohen’s/weighted/Fleiss’ kappa, Krippendorff’s alpha, ICC, and Gwet’s AC1. Use when raters coded the same cases and agreement must be reported。
如何安装评分者间信度分析?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「评分者间信度分析」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
评分者间信度分析需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
评分者间信度分析经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。