科研 Skill

内容分析编码方法

content-analysis-methodology

Designs codebooks, sampling plans, and intercoder-reliability checks for systematic content analysis. Use when measuring patterns in media coverage or validating an LLM classifier.

发布者 Codar安全审核通过v0.1.0更新于 2026-11-18

SKILL.md

Content analysis methodology

A systematic research method for measuring patterns in media content, text, images, or video at scale, the core method of academic communication research and a useful complement to data-journalism's data-oriented methods when the "data" is unstructured text or media rather than a spreadsheet.

Purpose

Content analysis answers questions like "how often did cable news mention inflation in election years" or "do local TV crime stories disproportionately feature Black suspects relative to arrest-rate data" with a reproducible, quantifiable method rather than a reporter's impression. The method's credibility rests entirely on two things this skill is built around: a codebook precise enough that two independent coders agree most of the time, and a sampling plan that supports the claim being made.

When to use

  • Measuring how often, how prominently, or in what framing a topic, person, or group appears across a body of media content (articles, broadcasts, social posts, press releases).
  • Testing a specific hypothesis about bias, framing, or representation in coverage (e.g., source diversity, tone, topic emphasis) that needs to stand up to methodological scrutiny.
  • Academic or think-tank communication research requiring a citable, replicable method section.
  • Validating an AI/LLM-based classification pipeline against a human-coded gold standard before trusting it at scale.

When not to use

  • A single reporter's close reading of a handful of articles for a narrative story, that's ordinary reporting, not content analysis, and dressing it up with this method's vocabulary overstates its rigor.
  • When the underlying claim is really about structured data (contract awards, campaign spending) rather than about patterns in unstructured content, use data-journalism instead.

Step 1: Define the unit of analysis

Decide precisely what gets coded as one unit before sampling anything:

Unit typeExampleUse when
Whole story/articleOne newspaper articleMeasuring topic presence, overall tone
ParagraphOne paragraph within an articleFine-grained framing analysis
Sentence/clauseOne sentenceClaim-level or quote-level coding
Source/quoteEach quoted or named sourceSource diversity, who gets to speak
Image/frameOne photo, one video frame or shotVisual framing, representation analysis
SegmentOne broadcast segment or social postCross-platform comparison

Mixing units mid-study (coding some things by article, others by paragraph) breaks comparability. Pick one level and code everything at that level.

Step 2: Define the population and draw the sample

  • Population: the full set of content the claim is about (e.g., "all evening broadcasts on the three major US networks during the 2026 election cycle"). State it precisely enough that someone else could reconstruct it.
  • Sampling frame: the actual retrievable source (a news database like Nexis Uni, GDELT, a network's own archive, a platform API) — note where the frame diverges from the true population (e.g., a paywalled archive that excludes some outlets, a platform API that doesn't return deleted posts).
  • Sampling method: census (code everything, feasible only for small populations), simple random sample, or constructed-week sampling (a classic technique for news content: sample one Monday, one Tuesday, etc. from different weeks to control for day-of-week publishing patterns, more efficient than pure random sampling for periodicity-heavy content).
  • Sample size: there's no universal rule, but report the margin of error or confidence interval for any population-level claim based on a sample, and state the final N actually coded (after excluding duplicates, irrelevant hits, and paywalled/inaccessible items).

Step 3: Build the codebook

The codebook is the single most important artifact, it is what makes the study replicable. For each variable:

## Variable: [name]

**Definition:** [precise, operational definition — not a vague concept]
**Values:** [exhaustive, mutually exclusive category list, or numeric scale]
**Decision rules:** [how to handle ambiguous cases, with examples]
**Examples:**
  - [example unit] → coded as [value], because [reasoning]
  - [borderline example] → coded as [value], because [reasoning, explicitly
    addressing why it's not the other plausible value]

Good codebooks spend most of their length on decision rules for borderline cases, not on the easy cases. If coders keep disagreeing on a particular type of case during pilot coding, that's a signal the codebook needs a new rule, not that the coders need more training.

Pilot the codebook on a small sample (20–50 units) with at least two coders, revise based on where they disagree, then freeze it before the real coding begins. Changing the codebook mid-study invalidates earlier coding.

Step 4: Establish intercoder reliability

Before trusting single-coder results (or an LLM classifier) at scale, measure agreement between two or more independent coders (or a coder and a validated gold standard) on a shared subsample, typically 10–20% of the full sample.

Why simple percent agreement is misleading: if 90% of content falls into one category, two coders who both just guess that category will "agree" 81% of the time by chance alone. Use a chance-corrected statistic instead:

StatisticHandlesCommon threshold
Cohen's kappaTwo coders, nominal categoriesκ ≥ 0.61 "substantial", ≥ 0.80 "excellent" (Landis & Koch 1977 scale, widely cited but treat as a rough guide, not a law)
Krippendorff's alphaAny number of coders, nominal/ordinal/interval/ratio data, missing dataα ≥ 0.80 for confident conclusions, 0.667 the lowest defensible floor for tentative conclusions (Krippendorff's own thresholds, widely adopted in communication research)
Scott's piTwo coders, nominal, assumes same marginal distributionRarely preferred over kappa/alpha today

Tools:

  • R: irr package (kappa2(), kripp.alpha()), or DescTools.
  • Python: nltk.metrics.agreement.AnnotationTask (for kappa/alpha), krippendorff package (pip install krippendorff) for a dedicated, well-tested Krippendorff's alpha implementation that handles missing data.
  • Report per-variable reliability, not just one overall number, a codebook with ten variables can have excellent reliability on easy variables (does the story mention a named topic) and poor reliability on hard ones (overall tone), and readers need to know which is which.

Step 5: Code the full sample and analyze

  • Once reliability is established on the pilot subsample, code the remainder (single-coder is standard practice at this stage if reliability was validated; double-code a rotating subsample throughout the full study to catch coder drift over a long project).
  • Analyze with the unit of analysis as the unit of statistical analysis, not the raw sentence count, if the unit is "story," run statistics on per-story codes, don't pool to the sentence level without re-justifying the unit.
  • Report frequencies, cross-tabulations, and, where a comparison is being made (e.g., outlet A vs. outlet B), the appropriate test (chi-square for categorical comparisons, t-test/ANOVA for means) with effect sizes, not just p-values.

Using LLMs for content analysis (and validating them)

LLMs can code large corpora faster than human coders, but every automated classification pipeline needs the same reliability validation as a human coder:

  1. Hand-code a gold-standard subsample (100+ units) using the frozen codebook.
  2. Run the LLM classifier on the same subsample with the codebook definitions in the prompt.
  3. Compute agreement (kappa or alpha) between the LLM output and the human gold standard, using the same thresholds as Step 4.
  4. Only proceed to full-corpus LLM coding if agreement clears the same bar you'd require of a human coder; report the validation statistic in any published methodology, not just "we used an LLM to code this."
  5. Re-validate on a fresh subsample if the prompt, model, or corpus changes materially, a validated pipeline is validated for that configuration, not forever.

Quality checks

  • Unit of analysis defined and held constant throughout.
  • Population and sampling frame explicitly stated, including known gaps.
  • Codebook frozen before full coding began, with a version/date on it.
  • Intercoder reliability reported per variable, using a chance-corrected statistic (kappa or alpha), on an independently coded subsample.
  • If an LLM did the coding, its agreement with a human gold standard is reported using the same statistic and threshold as a human coder would need to clear.
  • Final N reported after exclusions, with reasons for exclusion stated.
  • Any comparative claim (outlet A covers X more than outlet B) backed by a statistical test, not just a raw count difference.

Prerequisites

  • Requires the content corpus to be legally and practically accessible (a news database subscription, platform API access, or an internal archive); budget time for retrieval before coding can start.
  • Requires at least one other person to serve as a second coder for the reliability check, a single-coder study with no reliability check is a much weaker form of evidence and should be labeled as such.
  • Basic statistics (chi-square, correlation, chance-corrected agreement) is needed to interpret and report results correctly.

Limitations

  • Content analysis measures manifest content (what's observably there), not reception or effects, a finding that a topic appears more often in one outlet doesn't by itself establish that it changed anyone's opinion. Don't overclaim effects from a pure content measure.
  • Reliability statistics measure whether coders agree with each other (or an LLM agrees with coders), not whether the codebook itself is theoretically valid, a high kappa with a poorly conceived codebook still produces a rigorous measurement of the wrong thing.
  • Constructed-week and other periodicity-aware sampling methods are tuned for traditional news-cycle content; they transfer imperfectly to platforms with different temporal dynamics (a viral social post's lifecycle doesn't follow a weekly news cadence), adapt the sampling design to the medium.

Related skills

  • data-journalism, Statistical analysis once content has been coded into structured data
  • social-media-intelligence, Collecting the social-media corpus this method would then code
  • academic-writing, Writing up methods and results for publication

示例提示词

帮我设计一套编码本并计算编码员间信度来分析这批报道样本。

所属插件

新闻传播学面向新闻传播研究与调查报道:事实核查与信源核验、数据新闻、信息公开申请、采访准备与转录、编辑流程与新闻文风、社交媒体情报、图片元数据、网页与数字存档、危机传播。

安装该插件会一并获得同领域的 19 个 Skill / MCP。

同插件的其他科研 Skill

常见问题

内容分析编码方法能做什么?

Designs codebooks, sampling plans, and intercoder-reliability checks for systematic content analysis. Use when measuring patterns in media coverage or validating an LLM classifier。

如何安装内容分析编码方法?

在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「内容分析编码方法」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。

内容分析编码方法需要哪些权限?

该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。

内容分析编码方法经过官方审核吗?

是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。