科研 Skill
缺失数据处理
missing-data-handling
Diagnoses missing-data mechanisms such as MCAR, MAR, or MNAR and applies multiple imputation or deletion with sensitivity analysis. Use when a dataset has missing values.
SKILL.md
Missing Data Handling
Purpose
Guides a principled workflow for dealing with missing values in a research dataset: diagnosing why data are missing, choosing a defensible handling strategy, implementing it correctly, and reporting it in a way that survives reviewer scrutiny. Missing-data handling is one of the most common silent sources of bias in quantitative research — ad hoc listwise deletion or mean imputation can change effect sizes, inflate or deflate standard errors, and introduce bias that is invisible in a single regression table.
Use this skill whenever a dataset arrives with missing values (survey non-response, attrition in a longitudinal study, sensor dropout, merged datasets with partial coverage) and before running any substantive analysis (regression, ANOVA, structural equation models, machine learning).
When to Use
- A dataset has
NA/ blank / sentinel values in one or more variables and the next step is a statistical model. - A reviewer or advisor asks "how did you handle missing data?" and the honest answer is "I dropped the rows."
- Panel or longitudinal data has attrition and conclusions depend on whether dropout is related to the outcome.
- Deciding between complete-case analysis, imputation, and weighting approaches for a specific dataset.
Workflow
1. Quantify and visualize missingness
Before choosing a method, describe the missingness itself:
- Compute missingness rate per variable and per case (row).
- Build a missingness pattern table (which combinations of variables
are jointly missing, and how often) — in R,
naniar::vis_miss()ormice::md.pattern(); in Python,missingno.matrix()/missingno.heatmap(). - Flag variables with more than ~40-50% missing — these are usually better dropped or modeled separately than imputed.
- Check whether missingness clusters by time, site, device, or subgroup (e.g., all missing values come from one survey wave or one data collection site) — this is a strong clue about the mechanism.
2. Diagnose the mechanism (MCAR / MAR / MNAR)
The appropriate method depends on why data are missing, not just how much is missing:
- MCAR (Missing Completely At Random): missingness is unrelated to
observed or unobserved data. Testable (approximately) with Little's
MCAR test (
naniar::mcar_test()in R; note the test has low power and a non-significant result does not prove MCAR). If MCAR holds, complete-case analysis is unbiased (though inefficient). - MAR (Missing At Random): missingness depends on observed variables (e.g., older respondents are less likely to report income, but within age groups missingness is random). Diagnose by regressing a missingness indicator (0/1) on other observed variables — significant predictors suggest MAR rather than MCAR. Multiple imputation and full-information maximum likelihood (FIML) are valid under MAR.
- MNAR (Missing Not At Random): missingness depends on the unobserved value itself (e.g., people with the highest income are least likely to report it). MNAR cannot be verified from the observed data alone — it requires domain judgment, sensitivity analysis, or selection models (e.g., Heckman correction). State the MNAR risk explicitly rather than assuming it away.
Document this reasoning; a one-paragraph justification of the assumed mechanism is expected in most methods sections and almost always expected by a careful reviewer.
3. Choose a handling strategy
In roughly increasing order of sophistication and decreasing bias risk when MCAR does not hold:
- Listwise deletion (complete-case analysis): simplest, unbiased only under MCAR (or MAR when the missingness predictors are also included as covariates in the final model), loses power. Acceptable for low missingness (<5%) under plausible MCAR; otherwise should be reported as a sensitivity check, not the primary analysis.
- Pairwise deletion: uses all available pairs for each correlation/covariance; can produce non-positive-definite covariance matrices and inconsistent sample sizes across estimates. Avoid reporting as the primary method for multivariate models.
- Mean/median/mode imputation: attenuates variance and correlations; use only for quick exploratory checks, never for inferential analysis.
- Multiple imputation (MI): the default recommended approach for
MAR data with moderate missingness. Generate
m(typically 20-100, more if the fraction of missing information is high) complete datasets by drawing from the conditional distribution of each missing variable given the others, analyze each dataset separately, and pool estimates with Rubin's rules. In R:micepackage (mice(data, m = 20, method = "pmm")for mixed continuous/ categorical data,method = "logreg"/"polyreg"for binary/ categorical). In Python:sklearn.experimental.IterativeImputerorstatsmodels.imputation.mice.MICEData, ormiceforestfor a random-forest-based MICE implementation. - Full-information maximum likelihood (FIML): for SEM/latent
variable models, estimates parameters directly from all available
data without creating imputed datasets (
lavaanwithmissing = "fiml"in R;Mplus). - Pattern-mixture / selection models, inverse-probability weighting: for suspected MNAR or informative attrition in longitudinal data; requires an explicit model of the missingness process and should be treated as a sensitivity analysis, not a sole solution.
4. Implement multiple imputation correctly
Common implementation errors that a reviewer or a careful self-check should catch:
- Include the outcome variable in the imputation model when imputing predictors — excluding it biases the predictor-outcome relationship toward zero.
- Include all variables that will appear in the final analysis model (interactions, transformations) in the imputation model, not just the raw variables.
- Do not impute the outcome variable and then treat it as observed for model fit statistics that assume complete data, unless the analysis method is designed for it (e.g., Rubin's rules for the final estimates, not for descriptive fit indices computed on a single imputed dataset).
- Pool parameter estimates (not p-values alone) across the
mimputed datasets using Rubin's rules:pool()inmice,complete()+ manual pooling, orMIcombinein Python'sstatsmodels. - Report
m, the imputation method per variable, the number of iterations, and convergence diagnostics (trace plots of imputed values across iterations should show no systematic drift).
5. Sensitivity analysis and reporting
- Run the primary analysis under at least two assumptions (e.g., complete-case vs. multiple imputation) and report whether conclusions are stable. Divergent results are themselves a finding worth reporting.
- Report the missingness rate per key variable in a table (often Table 1 or an appendix table), the assumed mechanism, the method used, and software/package versions.
- For MNAR concerns, discuss direction and plausible magnitude of bias rather than ignoring the issue.
Prerequisites
- A dataset loaded in R or Python with missing values represented
consistently (
NA,NaN, or a documented sentinel value converted toNA/NaNbefore analysis). - Familiarity with the planned substantive analysis model, since the imputation model must include the same variables.
- R (
mice,naniar,lavaan) or Python (statsmodels,scikit-learn,miceforest,missingno) installed as needed — none of these are bundled with this skill; install via the project's usual package manager.
Limitations
- This skill provides a decision framework and standard commands, not a substitute for domain knowledge about why specific data are missing — that judgment call (especially MAR vs. MNAR) cannot be made from the data alone.
- Examples favor R's
miceand Python'sstatsmodels/scikit-learnecosystems; specialized designs (e.g., missing data in network data, image data, or time series with structural gaps) need method-specific references not covered here. - Multiple imputation assumes the imputation model is correctly specified; a badly misspecified imputation model (e.g., omitting an important interaction) can introduce new bias.
- Does not cover missing data in machine-learning pipelines where the deployment-time missingness mechanism differs from training-time missingness (a distinct problem requiring its own validation).
Quality Checklist
- Missingness rate and pattern reported per variable.
- Mechanism (MCAR/MAR/MNAR) stated with supporting reasoning, not just assumed.
- Chosen method matches the diagnosed mechanism and missingness rate.
- If multiple imputation:
m, method per variable, and pooling procedure documented; convergence checked. - At least one sensitivity analysis comparing methods reported.
- Outcome variable included in the imputation model when predictors are imputed.
示例提示词
我的数据集有很多缺失值,帮我判断缺失机制并选合适的插补方法。
所属插件
安装该插件会一并获得同领域的 54 个 Skill / MCP。
同插件的其他科研 Skill
- 智能体资源节约守则Applies token- and context-saving habits for tool choice, PDF handling, persistent memory, and privacy checks. Use at the start of sessions with many files or PDFs, or multi-session work.
- 英文论文双语精读器把英文论文转成中英对照精读档并生成离线双语阅读器,标注要点并输出读书笔记。精读单篇英文论文、要中英对照阅读时使用。
- 研究方法比较论证Compares quantitative, qualitative, and mixed research methods against what each can and cannot answer to build a methods table. Use when justifying a method choice in a thesis.
- 复杂系统分析方法Analyzes complex systems with power-law fitting, Hurst exponent, fractal dimension, agent-based modeling, and entropy measures. Use for nonlinear or complex-system data.
- 研究环境容器化部署Builds Docker and Singularity containers for reproducible computation and HPC deployment of research environments. Use when packaging an analysis pipeline for reuse or HPC.
- DVC数据版本管理Sets up DVC for dataset and pipeline versioning, remote storage, and experiment comparison in reproducible ML research workflows. Use when tracking large datasets or ML experiment runs.
- 期刊级统计图表绘制Creates publication-quality multi-panel figures with matplotlib, seaborn, or ggplot2 using colorblind-safe palettes and journal export settings. Use when preparing figures for a paper.
- 综述模块拆解规划Breaks a refined literature-review question into 3-5 non-overlapping modules to build a reading map, outline, or search plan. Use when structuring a literature review.
- 学位论文答辩教练模拟学位口试委员提问并提供承认回应证据三段式答辩框架与模拟问答。准备论文口试或研讨会问答环节时使用。
- 实验设计与样本量规划Designs rigorous experiments covering sample-size calculation, randomization strategy, pre-registration, and A/B test duration estimation. Use when planning an experiment or A/B test.
- 管理实验设计架构为管理与商学情境实验规划组间组内设计、对抗平衡与操弄检核,并产出设计书与情境材料模板。做商学实验设计时使用。
- 科研经费申请撰写Structures research grant proposals including NSF/NIH Specific Aims, budget justification, ERC narrative sections, and biosketch formatting. Use when drafting a funding application.
- 人类可读文档撰写规范Guides writing human-facing docs like README, PLAN, survey notes, and reviewer-response drafts with a clear two-tier structure. Use for text a human collaborator will read.
- 知识图谱与SPARQL检索Builds knowledge graphs and runs SPARQL queries over RDF and Wikidata for entity linking in research and library data. Use when modeling research entities as a linked graph.
- LaTeX论文写作与排版Writes, compiles, and submits LaTeX papers covering IMRaD structure, key packages, bibliography management, and arXiv preparation. Use when authoring or debugging a LaTeX manuscript.
- 文献比较矩阵构建器把一批PDF整理成含撤稿查核与APA7书目的文献资料库,并生成横向比较矩阵表。整理综述前要比较文献时使用。
- 跨库学术文献检索Searches OpenAlex, Semantic Scholar, and arXiv for academic literature with deduplication, citation snowballing, and BibTeX export. Use when collecting papers for a review or topic.
- 文献调研语料库维护Finds, verifies, and maintains a literature corpus from arXiv, OpenAlex, Crossref, and related databases with BibTeX entries and per-paper notes. Use when sourcing references for a paper.
- 文献综述研究缺口定位Converts a literature-review synthesis into a concrete, defensible research gap and future-research direction. Use when writing a conclusion or thesis contribution section.
- 文献矛盾结论梳理Analyzes conflicting findings across papers and explains disagreement by sample, method, or context differences for a stronger synthesis. Use when review sources disagree.
- 效应量汇总元分析Pools effect sizes across studies with fixed and random-effects models, heterogeneity tests, forest and funnel plots, and subgroup analysis. Use to synthesize quantitative study results.
- 混合方法研究设计Designs mixed-methods studies integrating qualitative coding, triangulation, content analysis, and QCA with inter-rater reliability checks. Use when combining qual and quant data.
- 科研机器学习流程Applies machine learning to research questions with scikit-learn pipelines, AutoML, SHAP explainability, and nested cross-validation. Use when building a predictive model for a study.
- 开放数据存储与FAIRDeposits and documents research data via Zenodo or OSF with FAIR assessment, data management plans, and DataCite metadata. Use when publishing or sharing a research dataset.
常见问题
缺失数据处理能做什么?
Diagnoses missing-data mechanisms such as MCAR, MAR, or MNAR and applies multiple imputation or deletion with sensitivity analysis. Use when a dataset has missing values。
如何安装缺失数据处理?
在科搭 Codar 左侧点击「插件」,在「组件」页签搜索「缺失数据处理」并安装;在 Claude Code 或 Codex 中可以发送本页的安装提示词,或下载 Skill 包后解压到 ~/.claude/skills 或 ~/.codex/skills。
缺失数据处理需要哪些权限?
该科研 Skill没有声明额外权限;执行具体操作时仍由 Agent 按自身规则请求确认。
缺失数据处理经过官方审核吗?
是。SpeedSkills 收录的每个版本都会记录内容摘要与权限声明,当前版本已通过安全审核。