科研 Skill

评分者间信度分析

inter-rater-reliability

Computes inter-rater/inter-coder reliability with Cohen’s/weighted/Fleiss’ kappa, Krippendorff’s alpha, ICC, and Gwet’s AC1. Use when raters coded the same cases and agreement must be reported.

发布者 Codar安全审核通过v0.1.0更新于 2026-11-18

SKILL.md

Inter-Rater Reliability

Purpose

Before using human-coded or human-rated data (behavioral observation, clinical diagnosis, content-analysis codes, essay/video ratings) in any downstream analysis, quantify how much raters agree. Percent agreement alone is misleading because it does not correct for chance agreement. This skill picks the right chance-corrected statistic for the data type (nominal, ordinal, binary, continuous; two raters or many; complete or missing data) and reports it with the interpretation thresholds the field actually uses.

When to Use

  • Two or more coders independently rated/classified the same set of items, video clips, transcripts, or participants, and the write-up needs a reliability statistic before the "real" analysis.
  • A clinical or diagnostic measure was scored by more than one rater (e.g., two clinicians scoring the same interview) and you need to report diagnostic agreement.
  • A continuous rating (e.g., 1–7 Likert ratings of stimuli, essay scores) was produced by multiple judges and you need an ICC for reliability.
  • Checking whether a coding scheme is reliable enough to hand off to a single coder for the rest of the dataset.

Does NOT trigger

  • Scale/questionnaire internal-consistency reliability across items (Cronbach's alpha, omega) — use psychometrics or alterlab-sem-psychometrics.
  • Test-retest reliability of a single rater's scores over time — that is a different reliability question (stability, not agreement across raters); a simple correlation or ICC(3,1) design still applies but interpret it as temporal stability, not inter-rater agreement.

Choosing the Right Statistic

Data type# RatersSame raters for all items?Missing data?Use
Binary / nominal2YesNoCohen's kappa
Ordinal (e.g., severity 1–4)2YesNoWeighted kappa (linear or quadratic weights)
Nominal≥ 3YesNoFleiss' kappa
Any (nominal/ordinal/interval/ratio)≥ 2Not necessarilyYes, allowedKrippendorff's alpha
Continuous / interval rating≥ 2YesNoIntraclass correlation (ICC)
Any, with known prevalence/bias concerns2YesNoGwet's AC1 (robust to the "kappa paradox" when prevalence is extreme)

The kappa paradox: Cohen's/Fleiss' kappa can be low even when raw agreement is very high, if the trait being rated is rare or very common (skewed prevalence) or if raters are both highly accurate but the base rate is imbalanced. When this happens, report Gwet's AC1 alongside kappa and say explicitly that the low kappa is a prevalence artifact, not poor rater agreement — do not silently pick whichever statistic looks better.

Prerequisites

  • Python 3.10+ with pandas, numpy, scipy, statsmodels; pip install statsmodels pandas numpy scipy.
  • For Krippendorff's alpha: pip install krippendorff (pure-Python implementation that handles missing data and multiple metric types out of the box), or implement it by hand using references below if the package is unavailable.
  • For ICC: pingouin (pip install pingouin) has a ready-made pingouin.intraclass_corr that implements all six Shrout & Fleiss (1979) ICC forms — simpler and less error-prone than hand-rolling ANOVA sums of squares.
  • Know, before computing anything: how many raters, whether every rater rated every item (fully crossed) or only a subset (partially crossed/missing), and whether the rating scale is nominal, ordinal, or interval — this decides which row of the table above applies.

Workflow

Step 1 — Cohen's kappa and weighted kappa (2 raters, nominal/ordinal)

import numpy as np
import pandas as pd
from sklearn.metrics import cohen_kappa_score

# ratings: two columns, one row per item, categorical codes (strings or ints)
# df = pd.DataFrame({"rater_a": [...], "rater_b": [...]})

def cohens_kappa(df: pd.DataFrame, col_a: str, col_b: str, weights=None) -> dict:
    """
    weights: None for unweighted (nominal) kappa, 'linear' or 'quadratic'
    for ordinal categories (e.g., severity ratings on an ordered scale).
    """
    kappa = cohen_kappa_score(df[col_a], df[col_b], weights=weights)
    n = len(df)
    po = (df[col_a] == df[col_b]).mean()  # observed raw agreement
    return {"kappa": round(kappa, 3), "raw_agreement": round(po, 3), "n": n}

result = cohens_kappa(df, "rater_a", "rater_b")
print(f"Cohen's kappa = {result['kappa']}, raw agreement = {result['raw_agreement']:.1%}")

Interpretation (Landis & Koch, 1977 — widely cited but treat as a rough guide, not a hard cutoff): < 0 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect.

Step 2 — Fleiss' kappa (3+ raters, nominal, every item rated by the same number of raters)

from statsmodels.stats.inter_rater import fleiss_kappa, aggregate_raters

# long_df: one row per (item, rater) with a 'category' column of codes
agg_table, categories = aggregate_raters(long_df[["item_id", "rater_id", "category"]])
kappa = fleiss_kappa(agg_table, method="fleiss")
print(f"Fleiss' kappa = {kappa:.3f} across {agg_table.shape[0]} items, "
      f"{len(categories)} categories")

aggregate_raters expects raw (item × rater) long-format data and converts it to the (item × category count) matrix fleiss_kappa needs — do not build that count matrix by hand unless you have verified the category order.

Step 3 — Krippendorff's alpha (any data type, handles missing ratings)

import krippendorff
import numpy as np

# reliability_data: one row per rater, one column per item; np.nan for
# items a given rater did not rate. level_of_measurement must match the
# variable type or the result is meaningless.
alpha = krippendorff.alpha(
    reliability_data=reliability_matrix,  # shape (n_raters, n_items)
    level_of_measurement="nominal",  # or 'ordinal', 'interval', 'ratio'
)
print(f"Krippendorff's alpha = {alpha:.3f}")

Krippendorff's own guidance: alpha ≥ .800 reliable; .667–.800 tentative conclusions only; below .667 do not rely on the coding. These thresholds are stricter than the Landis & Koch kappa bands above — do not reuse one scale for the other statistic.

Step 4 — ICC for continuous ratings

import pingouin as pg

# long_df: one row per (item, rater) with a numeric 'rating' column
icc_table = pg.intraclass_corr(
    data=long_df, targets="item_id", raters="rater_id", ratings="rating"
)
print(icc_table.set_index("Type"))
# Report ICC(2,k) for "average of these specific raters generalizes to
# other items rated by these same raters" (consistency across *items*),
# or ICC(2,1) for the reliability of a SINGLE rater's score; choose the
# form that matches how the ratings will actually be used downstream.

Picking the wrong ICC form is the most common misreport in this area: ICC(2,1) answers "how reliable is one rater's score", ICC(2,k) answers "how reliable is the mean of all k raters' scores" — these can differ substantially and are not interchangeable.

Step 5 — Gwet's AC1 (robust alternative when kappa looks paradoxically low)

# Gwet's AC1 for two raters, nominal data (hand computation — no mainstream
# Python package; port from Gwet, 2008, or compute in R with irrCAC::gwet.ac1.raw())
def gwet_ac1(df: pd.DataFrame, col_a: str, col_b: str, categories: list) -> float:
    n = len(df)
    po = (df[col_a] == df[col_b]).mean()
    pi_bar = {
        c: ((df[col_a] == c).sum() + (df[col_b] == c).sum()) / (2 * n)
        for c in categories
    }
    q = len(categories)
    pe = sum(p * (1 - p) for p in pi_bar.values()) / (q - 1) if q > 1 else 0.0
    return (po - pe) / (1 - pe) if pe < 1 else float("nan")

ac1 = gwet_ac1(df, "rater_a", "rater_b", categories=sorted(df["rater_a"].unique()))
print(f"Gwet's AC1 = {ac1:.3f}")

If no vetted Python package is available in the environment for AC1, prefer running irrCAC::gwet.ac1.raw() in R and reporting which implementation was used, rather than trusting a from-scratch formula without cross-checking it against a worked example from Gwet (2008) or Wongpakaran et al. (2013).

Output Template

STATISTIC:     <Cohen's kappa / weighted kappa / Fleiss' kappa / Krippendorff's alpha / ICC(form) / Gwet's AC1>
VALUE:         <point estimate> (raw agreement = <%>, n items = <n>, n raters = <k>)
INTERPRETATION: <qualitative band, with the specific guideline cited>
CAVEATS:       <prevalence/kappa-paradox check; missing-data handling; which ICC form and why>

Limitations

  • Chance-corrected statistics (kappa family) assume raters are independent; if raters discussed cases or one rater's codes were used to train the other, the statistic overstates true reliability.
  • Kappa/Fleiss' kappa require every item to have the same, fixed set of possible categories known in advance; novel categories appearing in only one rater's codes will distort the marginal proportions used in the chance-agreement term.
  • These statistics quantify agreement, not accuracy — high agreement between two biased raters does not mean the codes are correct against any external gold standard.
  • Small numbers of items (rule of thumb: under ~20–30) give kappa/alpha wide confidence intervals; report a bootstrap or Krippendorff's own CI procedure rather than a bare point estimate when n is small.

Quality Checks

  • Report raw percent agreement alongside the chance-corrected statistic so a reader can judge whether a low kappa reflects a prevalence artifact.
  • State the exact ICC form (e.g., ICC(2,1) vs ICC(2,k)) and the one-way/two-way, random/fixed model assumptions used — "the ICC was .81" alone is not reproducible.
  • When reliability is below the field's working threshold (e.g., kappa < .60, alpha < .667), recommend recoding a sample, refining the codebook, or retraining coders before proceeding to substantive analysis — do not silently continue with unreliable codes.
  • Verify the category set used for chance-agreement calculations matches across all raters; a category one rater never used should still appear in the denominator if it was a codebook option.

References

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
  • Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
  • Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage.
  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
  • Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.
  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

示例提示词

我想算两位编码员对这批访谈记录打分的Cohen's kappa和一致性。

所属插件

心理学面向心理学研究:心理测量与量表修订、结构方程、多层模型、反应时与眼动分析、PsychoPy 实验、功效分析与心理学元分析。

安装该插件会一并获得同领域的 13 个 Skill / MCP。

同插件的其他科研 Skill

常见问题

评分者间信度分析能做什么?

Computes inter-rater/inter-coder reliability with Cohen’s/weighted/Fleiss’ kappa, Krippendorff’s alpha, ICC, and Gwet’s AC1. Use when raters coded the same cases and agreement must be reported。

如何安装评分者间信度分析?

在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「评分者间信度分析」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。

评分者间信度分析需要哪些权限?

该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。

评分者间信度分析经过官方审核吗?

是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。