科研 Skill

缺失数据处理

missing-data-handling

Diagnoses missing-data mechanisms such as MCAR, MAR, or MNAR and applies multiple imputation or deletion with sensitivity analysis. Use when a dataset has missing values.

发布者 Codar安全审核通过v0.1.0更新于 2026-11-18

SKILL.md

Missing Data Handling

Purpose

Guides a principled workflow for dealing with missing values in a research dataset: diagnosing why data are missing, choosing a defensible handling strategy, implementing it correctly, and reporting it in a way that survives reviewer scrutiny. Missing-data handling is one of the most common silent sources of bias in quantitative research — ad hoc listwise deletion or mean imputation can change effect sizes, inflate or deflate standard errors, and introduce bias that is invisible in a single regression table.

Use this skill whenever a dataset arrives with missing values (survey non-response, attrition in a longitudinal study, sensor dropout, merged datasets with partial coverage) and before running any substantive analysis (regression, ANOVA, structural equation models, machine learning).

When to Use

  • A dataset has NA / blank / sentinel values in one or more variables and the next step is a statistical model.
  • A reviewer or advisor asks "how did you handle missing data?" and the honest answer is "I dropped the rows."
  • Panel or longitudinal data has attrition and conclusions depend on whether dropout is related to the outcome.
  • Deciding between complete-case analysis, imputation, and weighting approaches for a specific dataset.

Workflow

1. Quantify and visualize missingness

Before choosing a method, describe the missingness itself:

  • Compute missingness rate per variable and per case (row).
  • Build a missingness pattern table (which combinations of variables are jointly missing, and how often) — in R, naniar::vis_miss() or mice::md.pattern(); in Python, missingno.matrix() / missingno.heatmap().
  • Flag variables with more than ~40-50% missing — these are usually better dropped or modeled separately than imputed.
  • Check whether missingness clusters by time, site, device, or subgroup (e.g., all missing values come from one survey wave or one data collection site) — this is a strong clue about the mechanism.

2. Diagnose the mechanism (MCAR / MAR / MNAR)

The appropriate method depends on why data are missing, not just how much is missing:

  • MCAR (Missing Completely At Random): missingness is unrelated to observed or unobserved data. Testable (approximately) with Little's MCAR test (naniar::mcar_test() in R; note the test has low power and a non-significant result does not prove MCAR). If MCAR holds, complete-case analysis is unbiased (though inefficient).
  • MAR (Missing At Random): missingness depends on observed variables (e.g., older respondents are less likely to report income, but within age groups missingness is random). Diagnose by regressing a missingness indicator (0/1) on other observed variables — significant predictors suggest MAR rather than MCAR. Multiple imputation and full-information maximum likelihood (FIML) are valid under MAR.
  • MNAR (Missing Not At Random): missingness depends on the unobserved value itself (e.g., people with the highest income are least likely to report it). MNAR cannot be verified from the observed data alone — it requires domain judgment, sensitivity analysis, or selection models (e.g., Heckman correction). State the MNAR risk explicitly rather than assuming it away.

Document this reasoning; a one-paragraph justification of the assumed mechanism is expected in most methods sections and almost always expected by a careful reviewer.

3. Choose a handling strategy

In roughly increasing order of sophistication and decreasing bias risk when MCAR does not hold:

  • Listwise deletion (complete-case analysis): simplest, unbiased only under MCAR (or MAR when the missingness predictors are also included as covariates in the final model), loses power. Acceptable for low missingness (<5%) under plausible MCAR; otherwise should be reported as a sensitivity check, not the primary analysis.
  • Pairwise deletion: uses all available pairs for each correlation/covariance; can produce non-positive-definite covariance matrices and inconsistent sample sizes across estimates. Avoid reporting as the primary method for multivariate models.
  • Mean/median/mode imputation: attenuates variance and correlations; use only for quick exploratory checks, never for inferential analysis.
  • Multiple imputation (MI): the default recommended approach for MAR data with moderate missingness. Generate m (typically 20-100, more if the fraction of missing information is high) complete datasets by drawing from the conditional distribution of each missing variable given the others, analyze each dataset separately, and pool estimates with Rubin's rules. In R: mice package (mice(data, m = 20, method = "pmm") for mixed continuous/ categorical data, method = "logreg"/"polyreg" for binary/ categorical). In Python: sklearn.experimental.IterativeImputer or statsmodels.imputation.mice.MICEData, or miceforest for a random-forest-based MICE implementation.
  • Full-information maximum likelihood (FIML): for SEM/latent variable models, estimates parameters directly from all available data without creating imputed datasets (lavaan with missing = "fiml" in R; Mplus).
  • Pattern-mixture / selection models, inverse-probability weighting: for suspected MNAR or informative attrition in longitudinal data; requires an explicit model of the missingness process and should be treated as a sensitivity analysis, not a sole solution.

4. Implement multiple imputation correctly

Common implementation errors that a reviewer or a careful self-check should catch:

  • Include the outcome variable in the imputation model when imputing predictors — excluding it biases the predictor-outcome relationship toward zero.
  • Include all variables that will appear in the final analysis model (interactions, transformations) in the imputation model, not just the raw variables.
  • Do not impute the outcome variable and then treat it as observed for model fit statistics that assume complete data, unless the analysis method is designed for it (e.g., Rubin's rules for the final estimates, not for descriptive fit indices computed on a single imputed dataset).
  • Pool parameter estimates (not p-values alone) across the m imputed datasets using Rubin's rules: pool() in mice, complete() + manual pooling, or MIcombine in Python's statsmodels.
  • Report m, the imputation method per variable, the number of iterations, and convergence diagnostics (trace plots of imputed values across iterations should show no systematic drift).

5. Sensitivity analysis and reporting

  • Run the primary analysis under at least two assumptions (e.g., complete-case vs. multiple imputation) and report whether conclusions are stable. Divergent results are themselves a finding worth reporting.
  • Report the missingness rate per key variable in a table (often Table 1 or an appendix table), the assumed mechanism, the method used, and software/package versions.
  • For MNAR concerns, discuss direction and plausible magnitude of bias rather than ignoring the issue.

Prerequisites

  • A dataset loaded in R or Python with missing values represented consistently (NA, NaN, or a documented sentinel value converted to NA/NaN before analysis).
  • Familiarity with the planned substantive analysis model, since the imputation model must include the same variables.
  • R (mice, naniar, lavaan) or Python (statsmodels, scikit-learn, miceforest, missingno) installed as needed — none of these are bundled with this skill; install via the project's usual package manager.

Limitations

  • This skill provides a decision framework and standard commands, not a substitute for domain knowledge about why specific data are missing — that judgment call (especially MAR vs. MNAR) cannot be made from the data alone.
  • Examples favor R's mice and Python's statsmodels/scikit-learn ecosystems; specialized designs (e.g., missing data in network data, image data, or time series with structural gaps) need method-specific references not covered here.
  • Multiple imputation assumes the imputation model is correctly specified; a badly misspecified imputation model (e.g., omitting an important interaction) can introduce new bias.
  • Does not cover missing data in machine-learning pipelines where the deployment-time missingness mechanism differs from training-time missingness (a distinct problem requiring its own validation).

Quality Checklist

  • Missingness rate and pattern reported per variable.
  • Mechanism (MCAR/MAR/MNAR) stated with supporting reasoning, not just assumed.
  • Chosen method matches the diagnosed mechanism and missingness rate.
  • If multiple imputation: m, method per variable, and pooling procedure documented; convergence checked.
  • At least one sensitivity analysis comparing methods reported.
  • Outcome variable included in the imputation model when predictors are imputed.

示例提示词

我的数据集有很多缺失值,帮我判断缺失机制并选合适的插补方法。

所属插件

跨学科科研方法跨学科通用科研方法:系统综述与元分析、预注册、实验与问卷设计、文献矛盾与研究空白、科学计量与专利分析、可复现研究、答辩与审稿回复。

安装该插件会一并获得同领域的 54 个 Skill / MCP。

同插件的其他科研 Skill

常见问题

缺失数据处理能做什么?

Diagnoses missing-data mechanisms such as MCAR, MAR, or MNAR and applies multiple imputation or deletion with sensitivity analysis. Use when a dataset has missing values。

如何安装缺失数据处理?

在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「缺失数据处理」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。

缺失数据处理需要哪些权限?

该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。

缺失数据处理经过官方审核吗?

是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。